Methods and systems for three-dimensional model sharing

ABSTRACT

Examples of the disclosure describe systems and methods for decomposing and sharing 3D models. In an example method, a first version of a virtual three-dimensional model is displayed via a display of a wearable head device. A request is made to a host device for data associated with a second version of the virtual three-dimensional model, wherein the second version of the virtual three-dimensional model comprises a constituent part. It is determined whether the first version of the virtual three-dimensional model comprises the constituent part. In accordance with a determination that the first version of the virtual three-dimensional model does not comprise the constituent part, a request is made to the host device for data associated with the constituent part. The second version of the virtual three-dimensional model is displayed, via the display of the wearable head device. In accordance with a determination that the first version of the virtual three-dimensional model comprises the constituent part, a request is not made to the host device for data associated with the constituent part.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a Continuation of U.S. application Ser. No.16/581,716, filed Sep. 24, 2019, which claims priority to U.S.Provisional Application No. 62/735,675, filed on Sep. 24, 2018, thecontents of which are incorporated by reference herein in theirentirety.

FIELD

The present disclosure relates to sharing three-dimensional modelsbetween two or more computing systems, including mixed reality, imagingand visualization systems.

BACKGROUND

Modern computing and display technologies have facilitated thedevelopment of systems for so called “virtual reality,” “augmentedreality,” and “mixed reality” experiences, wherein digitally reproducedimages are presented to a user in a manner such that they seem to be, ormay be perceived as, real. A virtual reality (VR) scenario typicallyinvolves presentation of computer-generated virtual image informationwithout transparency to other actual real-world visual input. Anaugmented reality (AR) scenario typically involves presentation ofvirtual image information as an augmentation to visualization of theactual world around the user. Mixed reality (MR) is a type of augmentedreality in which physical and virtual objects may co-exist and interactin real time. Systems and methods disclosed herein address variouschallenges related to VR, AR and MR technology.

SUMMARY

Examples of the disclosure describe systems and methods for decomposingand sharing 3D models. In an example method, a first version of avirtual three-dimensional model is displayed via a display of a wearablehead device. A request is made to a host device for data associated witha second version of the virtual three-dimensional model, wherein thesecond version of the virtual three-dimensional model comprises aconstituent part. It is determined whether the first version of thevirtual three-dimensional model comprises the constituent part. Inaccordance with a determination that the first version of the virtualthree-dimensional model does not comprise the constituent part, arequest is made to the host device for data associated with theconstituent part. The second version of the virtual three-dimensionalmodel is displayed, via the display of the wearable head device. Inaccordance with a determination that the first version of the virtualthree-dimensional model comprises the constituent part, a request is notmade to the host device for data associated with the constituent part.

BRIEF DESCRIPTION OF THE DRAWINGS

Details of one or more implementations of the subject matter describedin this specification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages will becomeapparent from the description, the drawings, and the claims.

FIG. 1 depicts an illustration of a mixed reality scenario with certainvirtual reality objects, and certain physical objects viewed by aperson.

FIG. 2 schematically illustrates an example of a wearable system.

FIG. 3 schematically illustrates example components of a wearablesystem.

FIG. 4 schematically illustrates an example of a waveguide stack of awearable device for outputting image information to a user.

FIG. 5 is a process flow diagram of an example of a method forinteracting with a virtual user interface.

FIG. 6A is a block diagram of another example of a wearable system whichcan comprise an avatar processing and rendering system.

FIG. 6B illustrates example components of an avatar processing andrendering system.

FIG. 7 is a block diagram of an example of a wearable system includingvarious inputs into the wearable system.

FIG. 8 is a process flow diagram of an example of a method of renderingvirtual content in relation to recognized objects.

FIG. 9 schematically illustrates an overall system view depictingmultiple wearable systems interacting with each other.

FIG. 10 illustrates an example process of sharing 3D assets using thesystem and methods described herein.

FIG. 11 illustrates an example 3D model sharing system configuration forsharing 3D assets using the system and methods described herein.

FIG. 12 illustrates an example 3D model sharing process between a serverand client using the system and methods described herein.

FIG. 13 illustrates an example 3D model sharing system configuration forsharing 3D assets using the system and methods described herein.

FIG. 14 illustrates an example process for decomposing a full 3D modelusing the system and methods described herein.

FIG. 15 illustrates an example full 3D model using the system andmethods described herein.

FIG. 16 illustrates an example set of libraries utilized in a 3D modelsharing application to store constituent parts of a 3D model using thesystem and methods described herein.

FIG. 17 illustrates an example process for recomposing a full 3D modelfrom its constituent parts using the system and methods describedherein.

FIG. 18 illustrates an example full 3D model using the system andmethods described herein.

FIGS. 19A-19C illustrate an example mixed reality environment.

Throughout the drawings, reference numbers may be re-used to indicatecorrespondence between referenced elements. The drawings are provided toillustrate example embodiments described herein and are not intended tolimit the scope of the disclosure.

DETAILED DESCRIPTION

Mixed Reality Environment

Like all people, a user of a mixed reality system exists in a realenvironment—that is, a three-dimensional portion of the “real world,”and all of its contents, that are perceptible by the user. For example,a user perceives a real environment using one's ordinary humansenses—sight, sound, touch, taste, smell—and interacts with the realenvironment by moving one's own body in the real environment. Locationsin a real environment can be described as coordinates in a coordinatespace; for example, a coordinate can comprise latitude, longitude, andelevation with respect to sea level; distances in three orthogonaldimensions from a reference point; or other suitable values. Likewise, avector can describe a quantity having a direction and a magnitude in thecoordinate space.

A computing device can maintain, for example in a memory associated withthe device, a representation of a virtual environment. As used herein, avirtual environment is a computational representation of athree-dimensional space. A virtual environment can includerepresentations of any object, action, signal, parameter, coordinate,vector, or other characteristic associated with that space. In someexamples, circuitry (e.g., a processor) of a computing device canmaintain and update a state of a virtual environment; that is, aprocessor can determine at a first time t0, based on data associatedwith the virtual environment and/or input provided by a user, a state ofthe virtual environment at a second time t1. For instance, if an objectin the virtual environment is located at a first coordinate at time t0,and has certain programmed physical parameters (e.g., mass, coefficientof friction); and an input received from user indicates that a forceshould be applied to the object in a direction vector; the processor canapply laws of kinematics to determine a location of the object at timet1 using basic mechanics. The processor can use any suitable informationknown about the virtual environment, and/or any suitable input (e.g.,real world parameters), to determine a state of the virtual environmentat a time t1. In maintaining and updating a state of a virtualenvironment, the processor can execute any suitable software, includingsoftware relating to the creation and deletion of virtual objects in thevirtual environment; software (e.g., scripts) for defining behavior ofvirtual objects or characters in the virtual environment; software fordefining the behavior of signals (e.g., audio signals) in the virtualenvironment; software for creating and updating parameters associatedwith the virtual environment; software for generating audio signals inthe virtual environment; software for handling input and output;software for implementing network operations; software for applyingasset data (e.g., animation data to move a virtual object over time); ormany other possibilities.

Output devices, such as a display or a speaker, can present any or allaspects of a virtual environment to a user. For example, a virtualenvironment may include virtual objects (which may includerepresentations of inanimate objects; people; animals; lights; etc.)that may be presented to a user. A processor can determine a view of thevirtual environment (for example, corresponding to a “camera” with anorigin coordinate, a view axis, and a frustum); and render, to adisplay, a viewable scene of the virtual environment corresponding tothat view. Any suitable rendering technology may be used for thispurpose. In some examples, the viewable scene may include only somevirtual objects in the virtual environment, and exclude certain othervirtual objects. Similarly, a virtual environment may include audioaspects that may be presented to a user as one or more audio signals.For instance, a virtual object in the virtual environment may generate asound originating from a location coordinate of the object (e.g., avirtual character may speak or cause a sound effect); or the virtualenvironment may be associated with musical cues or ambient sounds thatmay or may not be associated with a particular location. A processor candetermine an audio signal corresponding to a “listener” coordinate—forinstance, an audio signal corresponding to a composite of sounds in thevirtual environment, and mixed and processed to simulate an audio signalthat would be heard by a listener at the listener coordinate—and presentthe audio signal to a user via one or more speakers.

Because a virtual environment exists only as a computational structure,a user cannot directly perceive a virtual environment using one'sordinary senses. Instead, a user can perceive a virtual environment onlyindirectly, as presented to the user, for example by a display,speakers, haptic output devices, etc. Similarly, a user cannot directlytouch, manipulate, or otherwise interact with a virtual environment; butcan provide input data, via input devices or sensors, to a processorthat can use the device or sensor data to update the virtualenvironment. For example, a camera sensor can provide optical dataindicating that a user is trying to move an object in a virtualenvironment, and a processor can use that data to cause the object torespond accordingly in the virtual environment.

A mixed reality system can present to the user, for example using atransmissive display and/or one or more speakers (which may, forexample, be incorporated into a wearable head device), a mixed reality(“MR”) environment that combines aspects of a real environment and avirtual environment. In some embodiments, the one or more speakers maybe external to the head-mounted wearable unit. As used herein, an MRenvironment is a simultaneous representation of a real environment and acorresponding virtual environment. In some examples, the correspondingreal and virtual environments share a single coordinate space; in someexamples, a real coordinate space and one or more corresponding virtualcoordinate spaces are related to each other by one or moretransformation matrices (or other suitable representation). Accordingly,in some embodiments, a single coordinate (along with, in some examples,a transformation matrix) can define a first location in the realenvironment, and also a second, corresponding, location in the virtualenvironment; and vice versa.

In an MR environment, a virtual object (e.g., in a virtual environmentassociated with the MR environment) can correspond to a real object(e.g., in a real environment associated with the MR environment). Forinstance, if the real environment of an MR environment comprises a reallamp post (a real object) at a location coordinate, the virtualenvironment of the MR environment may comprise a corresponding virtuallamp post (a virtual object) at a corresponding location coordinate. Asused herein, the real object in combination with its correspondingvirtual object together constitute a “mixed reality object.” It is notnecessary for a virtual object to perfectly match or align with acorresponding real object. In some examples, a virtual object can be asimplified version of a corresponding real object. For instance, if areal environment includes a real lamp post, a corresponding virtualobject may comprise a cylinder of roughly the same height and radius asthe real lamp post (reflecting that lamp posts may be roughlycylindrical in shape). Simplifying virtual objects in this manner canallow computational efficiencies, and can simplify calculations to beperformed on such virtual objects. Further, in some examples of an MRenvironment, not all real objects in a real environment may beassociated with a corresponding virtual object. Likewise, in someexamples of an MR environment, not all virtual objects in a virtualenvironment may be associated with a corresponding real object. That is,some virtual objects may solely in a virtual environment of an MRenvironment, without any real-world counterpart. In some examples, notall real objects may be associated with a corresponding real object.

In some examples, virtual objects may have characteristics that differ,sometimes drastically, from those of corresponding real objects. Forinstance, while a real environment in an MR environment may comprise agreen, two-armed cactus—a prickly inanimate object—a correspondingvirtual object in the MR environment may have the characteristics of agreen, two-armed virtual character with human facial features and asurly demeanor. In this example, the virtual object resembles itscorresponding real object in certain characteristics (color, number ofarms); but differs from the real object in other characteristics (facialfeatures, personality). In this way, virtual objects have the potentialto represent real objects in a creative, abstract, exaggerated, orfanciful manner; or to impart behaviors (e.g., human personalities) tootherwise inanimate real objects. In some examples, virtual objects maybe purely fanciful creations with no real-world counterpart (e.g., avirtual monster in a virtual environment, perhaps at a locationcorresponding to an empty space in a real environment).

Compared to VR systems, which present the user with a virtualenvironment while obscuring the real environment, a mixed reality systempresenting an MR environment affords the advantage that the realenvironment remains perceptible while the virtual environment ispresented. Accordingly, the user of the mixed reality system is able touse visual and audio cues associated with the real environment toexperience and interact with the corresponding virtual environment. Asan example, while a user of VR systems may struggle to perceive orinteract with a virtual object displayed in a virtualenvironment—because, as noted above, a user cannot directly perceive orinteract with a virtual environment—a user of an MR system may find itintuitive and natural to interact with a virtual object by seeing,hearing, and touching a corresponding real object in his or her own realenvironment. This level of interactivity can heighten a user's feelingsof immersion, connection, and engagement with a virtual environment.Similarly, by simultaneously presenting a real environment and a virtualenvironment, mixed reality systems can reduce negative psychologicalfeelings (e.g., cognitive dissonance) and negative physical feelings(e.g., motion sickness) associated with VR systems. Mixed realitysystems further offer many possibilities for applications that mayaugment or alter our experiences of the real world.

FIG. 19A illustrates an example real environment 1900 in which a user1910 uses a mixed reality system 1912. Mixed reality system 1912 maycomprise a display (e.g., a transmissive display) and one or morespeakers, and one or more sensors (e.g., a camera), for example asdescribed below. The real environment 1900 shown comprises a rectangularroom 1904A, in which user 1910 is standing; and real objects 1922A (alamp), 1924A (a table), 1926A (a sofa), and 1928A (a painting). Room1904A further comprises a location coordinate 1906, which may beconsidered an origin of the real environment 1900. As shown in FIG. 19A,an environment/world coordinate system 1908 (comprising an x-axis 1908X,a y-axis 1908Y, and a z-axis 1908Z) with its origin at point 1906 (aworld coordinate), can define a coordinate space for real environment1900. In some embodiments, the origin point 1906 of theenvironment/world coordinate system 1908 may correspond to where themixed reality system 1912 was powered on. In some embodiments, theorigin point 1906 of the environment/world coordinate system 1908 may bereset during operation. In some examples, user 1910 may be considered areal object in real environment 1900; similarly, user 1910's body parts(e.g., hands, feet) may be considered real objects in real environment1900. In some examples, a user/listener/head coordinate system 1914(comprising an x-axis 1914X, a y-axis 1914Y, and a z-axis 1914Z) withits origin at point 1915 (e.g., user/listener/head coordinate) candefine a coordinate space for the user/listener/head on which the mixedreality system 1912 is located. The origin point 1915 of theuser/listener/head coordinate system 1914 may be defined relative to oneor more components of the mixed reality system 1912. For example, theorigin point 1915 of the user/listener/head coordinate system 1914 maybe defined relative to the display of the mixed reality system 1912 suchas during initial calibration of the mixed reality system 1912. A matrix(which may include a translation matrix and a Quaternion matrix or otherrotation matrix), or other suitable representation can characterize atransformation between the user/listener/head coordinate system 1914space and the environment/world coordinate system 1908 space. In someembodiments, a left ear coordinate 1916 and a right ear coordinate 1917may be defined relative to the origin point 1915 of theuser/listener/head coordinate system 1914. A matrix (which may include atranslation matrix and a Quaternion matrix or other rotation matrix), orother suitable representation can characterize a transformation betweenthe left ear coordinate 1916 and the right ear coordinate 1917, anduser/listener/head coordinate system 1914 space. The user/listener/headcoordinate system 1914 can simplify the representation of locationsrelative to the user's head, or to a head-mounted device, for example,relative to the environment/world coordinate system 1908. UsingSimultaneous Localization and Mapping (SLAM), visual odometry, or othertechniques, a transformation between user coordinate system 1914 andenvironment coordinate system 1908 can be determined and updated inreal-time.

FIG. 19B illustrates an example virtual environment 1930 thatcorresponds to real environment 1900. The virtual environment 1930 showncomprises a virtual rectangular room 1904B corresponding to realrectangular room 1904A; a virtual object 1922B corresponding to realobject 1922A; a virtual object 1924B corresponding to real object 1924A;and a virtual object 1926B corresponding to real object 1926A. Metadataassociated with the virtual objects 1922B, 1924B, 1926B can includeinformation derived from the corresponding real objects 1922A, 1924A,1926A. Virtual environment 1930 additionally comprises a virtual monster1932, which does not correspond to any real object in real environment1900. Real object 1928A in real environment 1900 does not correspond toany virtual object in virtual environment 1930. A persistent coordinatesystem 1933 (comprising an x-axis 1933X, a y-axis 1933Y, and a z-axis1933Z) with its origin at point 1934 (persistent coordinate), can definea coordinate space for virtual content. The origin point 1934 of thepersistent coordinate system 1933 may be defined relative/with respectto one or more real objects, such as the real object 1926A. A matrix(which may include a translation matrix and a Quaternion matrix or otherrotation matrix), or other suitable representation can characterize atransformation between the persistent coordinate system 1933 space andthe environment/world coordinate system 1908 space. In some embodiments,each of the virtual objects 1922B, 1924B, 1926B, and 1932 may have theirown persistent coordinate point relative to the origin point 1934 of thepersistent coordinate system 1933. In some embodiments, there may bemultiple persistent coordinate systems and each of the virtual objects1922B, 1924B, 1926B, and 1932 may have their own persistent coordinatepoint relative to one or more persistent coordinate systems.

With respect to FIGS. 19A and 19B, environment/world coordinate system1908 defines a shared coordinate space for both real environment 1900and virtual environment 1930. In the example shown, the coordinate spacehas its origin at point 1906. Further, the coordinate space is definedby the same three orthogonal axes (1908X, 1908Y, 1908Z). Accordingly, afirst location in real environment 1900, and a second, correspondinglocation in virtual environment 1930, can be described with respect tothe same coordinate space. This simplifies identifying and displayingcorresponding locations in real and virtual environments, because thesame coordinates can be used to identify both locations. However, insome examples, corresponding real and virtual environments need not usea shared coordinate space. For instance, in some examples (not shown), amatrix (which may include a translation matrix and a Quaternion matrixor other rotation matrix), or other suitable representation cancharacterize a transformation between a real environment coordinatespace and a virtual environment coordinate space.

FIG. 19C illustrates an example MR environment 1950 that simultaneouslypresents aspects of real environment 1900 and virtual environment 1930to user 1910 via mixed reality system 1912. In the example shown, MRenvironment 1950 simultaneously presents user 1910 with real objects1922A, 1924A, 1926A, and 1928A from real environment 1900 (e.g., via atransmissive portion of a display of mixed reality system 1912); andvirtual objects 1922B, 1924B, 1926B, and 1932 from virtual environment1930 (e.g., via an active display portion of the display of mixedreality system 1912). As above, origin point 1906 acts as an origin fora coordinate space corresponding to MR environment 1950, and coordinatesystem 1908 defines an x-axis, y-axis, and z-axis for the coordinatespace.

In the example shown, mixed reality objects comprise corresponding pairsof real objects and virtual objects (i.e., 1922A/1922B, 1924A/1924B,1926A/1926B) that occupy corresponding locations in coordinate space1908. In some examples, both the real objects and the virtual objectsmay be simultaneously visible to user 1910. This may be desirable in,for example, instances where the virtual object presents informationdesigned to augment a view of the corresponding real object (such as ina museum application where a virtual object presents the missing piecesof an ancient damaged sculpture). In some examples, the virtual objects(1922B, 1924B, and/or 1926B) may be displayed (e.g., via activepixelated occlusion using a pixelated occlusion shutter) so as toocclude the corresponding real objects (1922A, 1924A, and/or 1926A).This may be desirable in, for example, instances where the virtualobject acts as a visual replacement for the corresponding real object(such as in an interactive storytelling application where an inanimatereal object becomes a “living” character).

In some examples, real objects (e.g., 1922A, 1924A, 1926A) may beassociated with virtual content or helper data that may not necessarilyconstitute virtual objects. Virtual content or helper data canfacilitate processing or handling of virtual objects in the mixedreality environment. For example, such virtual content could includetwo-dimensional representations of corresponding real objects; customasset types associated with corresponding real objects; or statisticaldata associated with corresponding real objects. This information canenable or facilitate calculations involving a real object withoutincurring unnecessary computational overhead.

In some examples, the presentation described above may also incorporateaudio aspects. For instance, in MR environment 1950, virtual monster1932 could be associated with one or more audio signals, such as afootstep sound effect that is generated as the monster walks around MRenvironment 1950. As described further below, a processor of mixedreality system 1912 can compute an audio signal corresponding to a mixedand processed composite of all such sounds in MR environment 1950, andpresent the audio signal to user 1910 via one or more speakers includedin mixed reality system 1912 and/or one or more external speakers.

Mixed Reality Object Sharing

A 3D digital model of an object may be displayed as a virtual object toone or more real or digital people in an AR/VR/MR (hereinafter referredto as MR for simplicity) environment. For example, an automotiveengineer may wish to share a new car design with co-workers at theirweekly team meeting. Traditionally, design sharing may be accomplishedby providing several different views of the design so the viewer is ableto imagine the 3D object. In this case, the automotive engineer mayprint out perspective drawings to hand out to co-workers during themeeting. This may work for simple designs but it may be difficult forthe viewer to piece together the 3D object in the viewers mind if the 3Dobject has a complex shape or design. Alternatively or additionally,traditional methods of sharing a 3D design or model may involve creatinga physical prototype. In this example, the automotive engineer wouldneed to spend a time and money creating the physical prototype. Thephysical prototype may make it easier to understand the designer'sintent, but may only show a basic version of the design (viewer can onlysee the outside of the design but can't open it to view the internalcomponents, for example).

Embodiments of the disclosed systems and methods may provide forimproved 3D model sharing between computing systems. Continuing with theexample above, the present invention may enable an automotive engineerto create a new digital 3D model of a car and then quickly andefficiently share the model with co-workers during the meeting. Thepresent invention can also enable fast and easy unplanned sharing. Forexample, the automotive engineer's manager may wish to share a 3D designthe manager had previously designed in order to collaborate with theautomotive engineer. In some embodiments, the manager is able to accessthe old 3D car design and share with everyone present at the meetingwithout requiring people to download the design or click on a link toaccess.

Typically, in gaming for example, 3D models (alternatively called 3Dassets, or 3D content) are pre-loaded in the game application so when auser starts up the game, all of the 3D assets that will be viewed by theuser are already on the device. When updates are needed, the applicationwill add new content offline. For example, the game may apply a patchwhen the game is not running, and when the application is next opened,the new content is installed and ready for use. This system of sharingnew 3D models with two or more computing systems is convenient forinfrequent, planned updates, but is impractical when new 3D models arefrequently or unexpectedly required.

MR games or applications may be collaborative, where multiple peoplecontribute towards a common goal, such as an entire a group of studentslearning about the anatomy of a frog in school. If the teacher tries toexplain a concept and the students don't understand, it may be desirablefor the teacher to display a virtual 3D model of a frog to the studentsso they can see what the teacher is describing. Traditional systemswould require the students to download the 3D model or exit and re-enterthe application so the teacher can push the new frog model update to thestudents. Additionally, such methods may require large amounts of diskstorage in order to house all of the 3D asset data. An alternate methodof accessing 3D models without using disk space, may be desirable.

These problems may be solved by methods and systems for streaming dataover a network between a server and one or more clients. The server mayhave access to one or more 3D models and may have a 3D model sharingserver application downloaded and running. The 3D model sharingapplication may break down the model into constituent data parts,compress, package, and send the constituent data parts to clients on thenetwork. The client can receive the constituent parts during 3D modelsharing client application runtime, reconstruct the 3D model, and viewthe 3D model. In some embodiments, runtime may be when the game orapplication loop is operating, when the screen is displaying applicationcontent, and/or when the computing system is operating with theframework of the application. Any number of clients may be part of thenetwork. Any number of 3D models may be send to the clients. Two or moreclients may be one person with two or more devices (e.g. computingsystems) or may be two or more people each with one device.

An advantage of the present application is enabling 3D content sharingduring runtime.

The term host may be used interchangeably with server. The terms 3Dmodel, 3D asset, 3D content, and 3D model may be used interchangeably.

Examples of 3D Display of a Wearable System

A wearable system (also referred to herein as an augmented reality (AR)system) can be configured to present 2D or 3D virtual images to a user.The images may be still images, frames of a video, or a video, incombination or the like. At least a portion of the wearable system canbe implemented on a wearable device that can present a VR, AR, or MRenvironment, alone or in combination, for user interaction. The wearabledevice can be used interchangeably as an AR device (ARD). Further, forthe purpose of the present disclosure, the term “AR” is usedinterchangeably with the term “MR”.

FIG. 1 depicts an illustration of a mixed reality scenario with certainvirtual reality objects, and certain physical objects viewed by aperson. In FIG. 1 , an MR scene 100 is depicted wherein a user of an MRtechnology sees a real-world park-like setting 110 featuring people,trees, buildings in the background, and a concrete platform 120. Inaddition to these items, the user of the MR technology also perceivesthat he “sees” a robot statue 130 standing upon the real-world platform120, and a cartoon-like avatar character 140 flying by which seems to bea personification of a bumble bee, even though these elements do notexist in the real world.

In order for the 3D display to produce a true sensation of depth, andmore specifically, a simulated sensation of surface depth, it may bedesirable for each point in the display's visual field to generate anaccommodative response corresponding to its virtual depth. If theaccommodative response to a display point does not correspond to thevirtual depth of that point, as determined by the binocular depth cuesof convergence and stereopsis, the human eye may experience anaccommodation conflict, resulting in unstable imaging, harmful eyestrain, headaches, and, in the absence of accommodation information,almost a complete lack of surface depth.

VR, AR, and MR experiences can be provided by display systems havingdisplays in which images corresponding to a plurality of depth planesare provided to a viewer. The images may be different for each depthplane (e.g., provide slightly different presentations of a scene orobject) and may be separately focused by the viewer's eyes, therebyhelping to provide the user with depth cues based on the accommodationof the eye required to bring into focus different image features for thescene located on different depth plane or based on observing differentimage features on different depth planes being out of focus. Asdiscussed elsewhere herein, such depth cues can provide credibleperceptions of depth.

FIG. 2 illustrates an example of wearable system 200 which can beconfigured to provide an AR/VR/MR scene. The wearable system 200 canalso be referred to as the AR system 200. The wearable system 200 caninclude a display 220, and various mechanical and electronic modules andsystems to support the functioning of display 220. The display 220 maybe coupled to a frame 230, which is wearable by a user, wearer, orviewer 210. The display 220 can be positioned in front of the eyes ofthe user 210. The display 220 can present AR/VR/MR content to a user.The display 220 can comprise a head mounted display (HMD) that is wornon the head of the user.

In some embodiments, a speaker 240 can be coupled to the frame 230 andpositioned adjacent the ear canal of the user (in some embodiments,another speaker, not shown, can be positioned adjacent the other earcanal of the user to provide for stereo/shapeable sound control). Thedisplay 220 can include an audio sensor (e.g., a microphone) 232 fordetecting an audio stream from the environment and capture ambientsound. In some embodiments, one or more other audio sensors, not shown,can be positioned to provide stereo sound reception. Stereo soundreception can be used to determine the location of a sound source. Thewearable system 200 can perform voice or speech recognition on the audiostream.

The wearable system 200 can include an outward-facing imaging system 464(shown in FIG. 4 ) which observes the world in the environment aroundthe user. The wearable system 200 can also include an inward-facingimaging system 462 (shown in FIG. 4 ) which can track the eye movementsof the user. The inward-facing imaging system may track either one eye'smovements or both eyes' movements. The inward-facing imaging system 462may be attached to the frame 230 and may be in electrical communicationwith the processing modules 260 or 270, which may process imageinformation acquired by the inward-facing imaging system to determine,e.g., the pupil diameters or orientations of the eyes, eye movements oreye pose of the user 210. The inward-facing imaging system 462 mayinclude one or more cameras. For example, at least one camera may beused to image each eye. The images acquired by the cameras may be usedto determine pupil size or eye pose for each eye separately, therebyallowing presentation of image information to each eye to be dynamicallytailored to that eye.

As an example, the wearable system 200 can use the outward-facingimaging system 464 or the inward-facing imaging system 462 to acquireimages of a pose of the user. The images may be still images, frames ofa video, or a video.

The display 220 can be operatively coupled 250, such as by a wired leador wireless connectivity, to a local data processing module 260 whichmay be mounted in a variety of configurations, such as fixedly attachedto the frame 230, fixedly attached to a helmet or hat worn by the user,embedded in headphones, or otherwise removably attached to the user 210(e.g., in a backpack-style configuration, in a belt-coupling styleconfiguration).

The local processing and data module 260 may comprise a hardwareprocessor, as well as digital memory, such as non-volatile memory (e.g.,flash memory), both of which may be utilized to assist in theprocessing, caching, and storage of data. The data may include data a)captured from sensors (which may be, e.g., operatively coupled to theframe 230 or otherwise attached to the user 210), such as image capturedevices (e.g., cameras in the inward-facing imaging system or theoutward-facing imaging system), audio sensors (e.g., microphones),inertial measurement units (IMUs), accelerometers, compasses, globalpositioning system (GPS) units, radio devices, or gyroscopes; or b)acquired or processed using remote processing module 270 or remote datarepository 280, possibly for passage to the display 220 after suchprocessing or retrieval. The local processing and data module 260 may beoperatively coupled by communication links 262 or 264, such as via wiredor wireless communication links, to the remote processing module 270 orremote data repository 280 such that these remote modules can beavailable as resources to the local processing and data module 260. Inaddition, remote processing module 280 and remote data repository 280may be operatively coupled to each other.

In some embodiments, the remote processing module 270 may comprise oneor more processors configured to analyze and process data or imageinformation. In some embodiments, the remote data repository 280 maycomprise a digital data storage facility, which may be available throughthe internet or other networking configuration in a “cloud” resourceconfiguration. In some embodiments, all data can be stored and allcomputations can be performed in the local processing and data module,allowing fully autonomous use from a remote module.

Example Components of a Wearable System

FIG. 3 schematically illustrates example components of a wearablesystem. FIG. 3 shows a wearable system 200 which can include a display220 and a frame 230. A blown-up view 202 schematically illustratesvarious components of the wearable system 200. In certain implements,one or more of the components illustrated in FIG. 3 can be part of thedisplay 220. The various components alone or in combination can collecta variety of data (such as e.g., audio or visual data) associated withthe user of the wearable system 200 or the user's environment. It shouldbe appreciated that other embodiments may have additional or fewercomponents depending on the application for which the wearable system isused. Nevertheless, FIG. 3 provides a basic idea of some of the variouscomponents and types of data that may be collected, analyzed, and storedthrough the wearable system.

FIG. 3 shows an example wearable system 200 which can include thedisplay 220. The display 220 can comprise a display lens 226 that may bemounted to a user's head or a housing or frame 230, which corresponds tothe frame 230. The display lens 226 may comprise one or more transparentmirrors or diffractive optical elements positioned by the housing 230 infront of the user's eyes 302, 304 and may be configured to directprojected light 338 into the eyes 302, 304 and facilitate beam shaping,while also allowing for transmission of at least some light from thelocal environment. The wavefront of the projected light beam 338 maydiverge to coincide with a desired focal distance of the projectedlight. As illustrated, two wide-field-of-view machine vision cameras 316(also referred to as world cameras) can be coupled to the housing 230 toimage the environment around the user. These cameras 316 can be dualcapture visible light/non-visible (e.g., infrared) light cameras. Thecameras 316 may be part of the outward-facing imaging system 464 shownin FIG. 4 . Image acquired by the world cameras 316 can be processed bythe pose processor 336. For example, the pose processor 336 canimplement one or more object recognizers 708 (e.g., shown in FIG. 7 ) toidentify a pose of a user or another person in the user's environment orto identify a physical object in the user's environment.

With continued reference to FIG. 3 , a pair of light projector moduleswith display optics and lens configured to direct light 338 into theeyes 302, 304 are shown. The depicted view also shows two miniatureinfrared cameras 324 paired with infrared light (such as light emittingdiodes “LED”s), which can be configured to be able to track the eyes302, 304 of the user to support rendering and user input. The cameras324 may be part of the inward-facing imaging system 462 shown in FIG. 4The wearable system 200 can further feature a sensor assembly 339, whichmay comprise X, Y, and Z axis accelerometer capability as well as amagnetic compass and X, Y, and Z axis gyro capability, preferablyproviding data at a relatively high frequency, such as 200 Hz. Thesensor assembly 339 may be part of the IMU described with reference toFIG. 2A The depicted system 200 can also comprise a head pose processor336, such as an ASIC (application specific integrated circuit), FPGA(field programmable gate array), or ARM processor (advancedreduced-instruction-set machine), which may be configured to calculatereal or near-real time user head pose from wide field of view imageinformation output from the capture devices 316. The head pose processor336 can be a hardware processor and can be implemented as part of thelocal processing and data module 260 shown in FIG. 2A.

The wearable system can also include one or more depth sensors 234. Thedepth sensor 234 can be configured to measure the distance between anobject in an environment to a wearable device. The depth sensor 234 mayinclude a laser scanner (e.g., a lidar), an ultrasonic depth sensor, ora depth sensing camera. In certain implementations, where the cameras316 have depth sensing ability, the cameras 316 may also be consideredas depth sensors 234.

Also shown is a processor 332 which can be configured to execute digitalor analog processing to derive pose from the gyro, compass, oraccelerometer data from the sensor assembly 339. The processor 332 maybe part of the local processing and data module 260 shown in FIG. 2 .The wearable system 200 as shown in FIG. 3 can also include a positionsystem such as, e.g., a GPS 337 (global positioning system) to assistwith pose and positioning analyses. In addition, the GPS may furtherprovide remotely-based (e.g., cloud-based) information about the user'senvironment. This information may be used for recognizing objects orinformation in user's environment.

The wearable system may combine data acquired by the GPS 337 and aremote computing system (such as, e.g., the remote processing module270, another user's ARD, etc.) which can provide more information aboutthe user's environment. As one example, the wearable system candetermine the user's location based on GPS data and retrieve a world map(e.g., by communicating with a remote processing module 270) includingvirtual objects associated with the user's location. As another example,the wearable system 200 can monitor the environment using the worldcameras 316 (which may be part of the outward-facing imaging system 464shown in FIG. 4 ). Based on the images acquired by the world cameras316, the wearable system 200 can detect objects in the environment(e.g., by using one or more object recognizers 708 shown in FIG. 7 ).The wearable system can further use data acquired by the GPS 337 tointerpret the characters.

The wearable system 200 may also comprise a rendering engine 334 whichcan be configured to provide rendering information that is local to theuser to facilitate operation of the scanners and imaging into the eyesof the user, for the user's view of the world. The rendering engine 334may be implemented by a hardware processor (such as, e.g., a centralprocessing unit or a graphics processing unit). In some embodiments, therendering engine is part of the local processing and data module 260.The rendering engine 334 can be communicatively coupled (e.g., via wiredor wireless links) to other components of the wearable system 200. Forexample, the rendering engine 334, can be coupled to the eye cameras 324via communication link 274, and be coupled to a projecting subsystem 318(which can project light into user's eyes 302, 304 via a scanned laserarrangement in a manner similar to a retinal scanning display) via thecommunication link 272. The rendering engine 334 can also be incommunication with other processing units such as, e.g., the sensor poseprocessor 332 and the image pose processor 336 via links 276 and 294respectively.

The cameras 324 (e.g., mini infrared cameras) may be utilized to trackthe eye pose to support rendering and user input. Some example eye posesmay include where the user is looking or at what depth he or she isfocusing (which may be estimated with eye vergence). The GPS 337, gyros,compass, and accelerometers 339 may be utilized to provide coarse orfast pose estimates. One or more of the cameras 316 can acquire imagesand pose, which in conjunction with data from an associated cloudcomputing resource, may be utilized to map the local environment andshare user views with others.

The example components depicted in FIG. 3 are for illustration purposesonly. Multiple sensors and other functional modules are shown togetherfor ease of illustration and description. Some embodiments may includeonly one or a subset of these sensors or modules. Further, the locationsof these components are not limited to the positions depicted in FIG. 3. Some components may be mounted to or housed within other components,such as a belt-mounted component, a hand-held component, or a helmetcomponent. As one example, the image pose processor 336, sensor poseprocessor 332, and rendering engine 334 may be positioned in a beltpackand configured to communicate with other components of the wearablesystem via wireless communication, such as ultra-wideband, Wi-Fi,Bluetooth, etc., or via wired communication. The depicted housing 230preferably is head-mountable and wearable by the user. However, somecomponents of the wearable system 200 may be worn to other portions ofthe user's body. For example, the speaker 240 may be inserted into theears of a user to provide sound to the user.

Regarding the projection of light 338 into the eyes 302, 304 of theuser, in some embodiments, the cameras 324 may be utilized to measurewhere the centers of a user's eyes are geometrically verged to, which,in general, coincides with a position of focus, or “depth of focus”, ofthe eyes. A 3-dimensional surface of all points the eyes verge to can bereferred to as the “horopter”. The focal distance may take on a finitenumber of depths, or may be infinitely varying. Light projected from thevergence distance appears to be focused to the subject eye 302, 304,while light in front of or behind the vergence distance is blurred.

The human visual system is complicated and providing a realisticperception of depth is challenging. Viewers of an object may perceivethe object as being three-dimensional due to a combination of vergenceand accommodation. Vergence movements (e.g., rolling movements of thepupils toward or away from each other to converge the lines of sight ofthe eyes to fixate upon an object) of the two eyes relative to eachother are closely associated with focusing (or “accommodation”) of thelenses of the eyes. Under normal conditions, changing the focus of thelenses of the eyes, or accommodating the eyes, to change focus from oneobject to another object at a different distance will automaticallycause a matching change in vergence to the same distance, under arelationship known as the “accommodation-vergence reflex.” Likewise, achange in vergence will trigger a matching change in accommodation,under normal conditions. Display systems that provide a better matchbetween accommodation and vergence may form more realistic andcomfortable simulations of three-dimensional imagery.

Further spatially coherent light with a beam diameter of less than about0.7 millimeters can be correctly resolved by the human eye regardless ofwhere the eye focuses. Thus, to create an illusion of proper focaldepth, the eye vergence may be tracked with the cameras 324, and therendering engine 334 and projection subsystem 318 may be utilized torender all objects on or close to the vergence in focus, and all otherobjects at varying degrees of resolution. Preferably, the system 220renders to the user at a frame rate of about 60 frames per second orgreater. As described above, preferably, the cameras 324 may be utilizedfor eye tracking, and software may be configured to pick up not onlyvergence geometry but also focus location cues to serve as user inputs.Preferably, such a display system is configured with brightness andcontrast suitable for day or night use.

In some embodiments, the display system preferably has latency of lessthan about 20 milliseconds for visual object alignment, less than about0.1 degree of angular alignment, and about 1 arc minute of resolution,which, without being limited by theory, is believed to be approximatelythe limit of the human eye. The display system 220 may be integratedwith a localization system, which may involve GPS elements, opticaltracking, compass, accelerometers, or other data sources, to assist withposition and pose determination; localization information may beutilized to facilitate accurate rendering in the user's view of thepertinent world (e.g., such information would facilitate the glasses toknow where they are with respect to the real world).

In some embodiments, the wearable system 200 is configured to displayone or more virtual images based on the accommodation of the user'seyes. Unlike prior 3D display approaches that force the user to focuswhere the images are being projected, in some embodiments, the wearablesystem can be configured to automatically vary the focus of projectedvirtual content to allow for a more comfortable viewing of one or moreimages presented to the user. For example, if the user's eyes have acurrent focus of 1 m, the image may be projected to coincide with theuser's focus. If the user shifts focus to 3 m, the image can beprojected to coincide with the new focus. Thus, rather than forcing theuser to a predetermined focus, the wearable system 200 of someembodiments can allow the user's eye to a function in a more naturalmanner.

Such a wearable system 200 may eliminate or reduce the incidences of eyestrain, headaches, and other physiological symptoms typically observedwith respect to virtual reality devices. To achieve this, variousembodiments of the wearable system 200 can be configured to projectvirtual images at varying focal distances, through one or more variablefocus elements (VFEs). In one or more embodiments, 3D perception may beachieved through a multi-plane focus system that projects images atfixed focal planes away from the user. Other embodiments can employvariable plane focus, wherein the focal plane may move back and forth inthe z-direction to coincide with the user's present state of focus.

In both the multi-plane focus systems and variable plane focus systems,wearable system 200 may employ eye tracking to determine a vergence ofthe user's eyes, determine the user's current focus, and project thevirtual image at the determined focus. In other embodiments, wearablesystem 200 can comprise a light modulator that variably projects,through a fiber scanner, or other light generating source, light beamsof varying focus in a raster pattern across the retina. Thus, theability of the display of the wearable system 200 to project images atvarying focal distances not only eases accommodation for the user toview objects in 3D, but may also be used to compensate for user ocularanomalies. In some other embodiments, a spatial light modulator mayproject the images to the user through various optical components. Forexample, as described further below, the spatial light modulator mayproject the images onto one or more waveguides, which then transmit theimages to the user.

Waveguide Stack Assembly

FIG. 4 illustrates an example of a waveguide stack for outputting imageinformation to a user. A wearable system 400 can include a stack ofwaveguides, or stacked waveguide assembly 480 that may be utilized toprovide three-dimensional perception to the eye/brain using a pluralityof waveguides 432 b, 434 b, 436 b, 438 b, 4400 b. In some embodiments,the wearable system 400 may correspond to wearable system 200 of FIG. 2, with FIG. 4 schematically showing some parts of that wearable system200 in greater detail. For example, in some embodiments, the waveguideassembly 480 may be integrated into the display 220 of FIG. 2 .

With continued reference to FIG. 4 , the waveguide assembly 480 may alsoinclude a plurality of features 458, 456, 454, 452 between thewaveguides. In some embodiments, the features 458, 456, 454, 452 may belenses. In other embodiments, the features 458, 456, 454, 452 may not belenses. Rather, they may simply be spacers (e.g., cladding layers orstructures for forming air gaps).

The waveguides 432 b, 434 b, 436 b, 438 b, 440 b or the plurality oflenses 458, 456, 454, 452 may be configured to send image information tothe eye with various levels of wavefront curvature or light raydivergence. Each waveguide level may be associated with a particulardepth plane and may be configured to output image informationcorresponding to that depth plane. Image injection devices 420, 422,424, 426, 428 may be utilized to inject image information into thewaveguides 440 b, 438 b, 436 b, 434 b, 432 b, each of which may beconfigured to distribute incoming light across each respectivewaveguide, for output toward the eye 410. Light can exit an outputsurface of the image injection devices 420, 422, 424, 426, 428 and canbe injected into a corresponding input edge of the waveguides 440 b, 438b, 436 b, 434 b, 432 b. In some embodiments, a single beam of light(e.g., a collimated beam) may be injected into each waveguide to outputan entire field of sample exit pupil beams that can be directed towardthe eye 410 at particular angles (and amounts of divergence)corresponding to the depth plane associated with a particular waveguide.

In some embodiments, the image injection devices 420, 422, 424, 426, 428can be discrete displays that each produce image information forinjection into a corresponding waveguide 440 b, 438 b, 436 b, 434 b, 432b, respectively. In some other embodiments, the image injection devices420, 422, 424, 426, 428 can be the output ends of a single multiplexeddisplay which may, e.g., pipe image information via one or more opticalconduits (such as fiber optic cables) to each of the image injectiondevices 420, 422, 424, 426, 428.

A controller 460 can control the operation of the display and the imageinjection devices 420, 422, 424, 426, 428. The controller 460 caninclude programming (e.g., instructions in a non-transitorycomputer-readable medium) that regulates the timing and provision ofimage information to the waveguides 440 b, 438 b, 436 b, 434 b, 432 b.In some embodiments, the controller 460 may be a single integral device,or a distributed system connected by wired or wireless communicationchannels. The controller 460 may be part of the processing modules 260or 270 (illustrated in FIG. 2 ) in some embodiments.

The waveguides 440 b, 438 b, 436 b, 434 b, 432 b may be configured topropagate light within each respective waveguide by total internalreflection (TIR). The waveguides 440 b, 438 b, 436 b, 434 b, 432 b mayeach be planar or have another shape (e.g., curved), with major top andbottom surfaces and edges extending between those major top and bottomsurfaces. In the illustrated configuration, the waveguides 440 b, 438 b,436 b, 434 b, 432 b may each include optical elements 440 a, 438 a, 436a, 434 a, 432 a that can be configured to outcouple light out of awaveguide by diffracting or otherwise redirecting the light propagatingwithin each respective waveguide. Outcoupled light can be outputted bythe waveguide at locations at which the light propagating in thewaveguide strikes a light redirecting element, such as a diffractivegrating, for example. The optical elements (440 a, 438 a, 436 a, 434 a,432 a) may, for example, be reflective or diffractive optical features.While illustrated disposed at the bottom major surfaces of thewaveguides 440 b, 438 b, 436 b, 434 b, 432 b for ease of description anddrawing clarity, in some embodiments, the optical elements 440 a, 438 a,436 a, 434 a, 432 a may be disposed at the top or bottom major surfaces,or may be disposed directly in the volume of the waveguides 440 b, 438b, 436 b, 434 b, 432 b. In some embodiments, the optical elements 440 a,438 a, 436 a, 434 a, 432 a may be formed in a layer of material that isattached to a transparent substrate to form the waveguides 440 b, 438 b,436 b, 434 b, 432 b. In some other embodiments, the waveguides 440 b,438 b, 436 b, 434 b, 432 b may be a monolithic piece of material and theoptical elements 440 a, 438 a, 436 a, 434 a, 432 a may be formed on asurface or in the interior of that piece of material.

With continued reference to FIG. 4 , as discussed herein, in someembodiments, each waveguide 440 b, 438 b, 436 b, 434 b, 432 b can beconfigured to output light to form an image corresponding to aparticular depth plane. For example, the waveguide 432 b nearest the eyemay be configured to deliver collimated light, as injected into suchwaveguide 432 b, to the eye 410. The collimated light may berepresentative of the optical infinity focal plane. The next waveguideup 434 b may be configured to send out collimated light which passesthrough the first lens 452 (e.g., a negative lens) before it can reachthe eye 410. First lens 452 may be configured to create a slight convexwavefront curvature so that the eye/brain interprets light coming fromthat next waveguide up 434 b as coming from a first focal plane closerinward toward the eye 410 from optical infinity. Similarly, the third upwaveguide 436 b can pass its output light through both the first lens452 and second lens 454 before reaching the eye 410. The combinedoptical power of the first and second lenses 452 and 454 may beconfigured to create another incremental amount of wavefront curvatureso that the eye/brain interprets light coming from the third waveguide436 b as coming from a second focal plane that is even closer inwardtoward the person from optical infinity than was light from the nextwaveguide up 434 b.

The other waveguide layers (e.g., waveguides 438 b, 440 b) and lenses(e.g., lenses 456, 458) can be similarly configured, with the highestwaveguide 440 b in the stack sending its output through all of thelenses between it and the eye for an aggregate focal powerrepresentative of the closest focal plane to the person. To compensatefor the stack of lenses 458, 456, 454, 452 when viewing/interpretinglight coming from the world 470 on the other side of the stackedwaveguide assembly 480, a compensating lens layer 430 may be disposed atthe top of the stack to compensate for the aggregate power of the lensstack 458, 456, 454, 452 below. Such a configuration can provide as manyperceived focal planes as there are available waveguide/lens pairings.Both the light extracting optical elements of the waveguides and thefocusing aspects of the lenses may be static (e.g., not dynamic orelectro-active). In some alternative embodiments, either or both may bedynamic using electro-active features.

With continued reference to FIG. 4 , the light extracting opticalelements 440 a, 438 a, 436 a, 434 a, 432 a may be configured to bothredirect light out of their respective waveguides and to output thislight with the appropriate amount of divergence or collimation for aparticular depth plane associated with the waveguide. As a result,waveguides having different associated depth planes may have differentconfigurations of light extracting optical elements, which output lightwith a different amount of divergence depending on the associated depthplane. In some embodiments, as discussed herein, the light extractingoptical elements 440 a, 438 a, 436 a, 434 a, 432 a may be volumetric orsurface features, which may be configured to output light at specificangles. For example, the light extracting optical elements 440 a, 438 a,436 a, 434 a, 432 a may be volume holograms, surface holograms, and/ordiffraction gratings.

In some embodiments, the light extracting optical elements 440 a, 438 a,436 a, 434 a, 432 a can be diffractive features that form a diffractionpattern, or “diffractive optical element” (also referred to herein as a“DOE”). Preferably, the DOE has a relatively low diffraction efficiencyso that only a portion of the light of the beam is deflected away towardthe eye 410 with each intersection of the DOE, while the rest continuesto move through a waveguide via total internal reflection. The lightcarrying the image information can thus be divided into a number ofrelated exit beams that exit the waveguide at a multiplicity oflocations and the result is a fairly uniform pattern of exit emissiontoward the eye 304 for this particular collimated beam bouncing aroundwithin a waveguide.

In some embodiments, one or more DOEs may be switchable between “on”state in which they actively diffract, and “off” state in which they donot significantly diffract. For instance, a switchable DOE may comprisea layer of polymer dispersed liquid crystal, in which microdroplets cancomprise a diffraction pattern in a host medium, and the refractiveindex of the microdroplets can be switched to substantially match therefractive index of the host material (in which case the pattern doesnot appreciably diffract incident light) or the microdroplet can beswitched to an index that does not match that of the host medium (inwhich case the pattern actively diffracts incident light).

In some embodiments, the number and distribution of depth planes ordepth of field may be varied dynamically based on the pupil sizes ororientations of the eyes of the viewer. Depth of field may changeinversely with a viewer's pupil size. As a result, as the sizes of thepupils of the viewer's eyes decrease, the depth of field can increasesuch that one plane that is not discernible because the location of thatplane is beyond the depth of focus of the eye may become discernible andappear more in focus with reduction of pupil size and commensurate withthe increase in depth of field. Likewise, the number of spaced apartdepth planes used to present different images to the viewer may bedecreased with the decreased pupil size. For example, a viewer may notbe able to clearly perceive the details of both a first depth plane anda second depth plane at one pupil size without adjusting theaccommodation of the eye away from one depth plane and to the otherdepth plane. These two depth planes may, however, be sufficiently infocus at the same time to the user at another pupil size withoutchanging accommodation.

In some embodiments, the display system may vary the number ofwaveguides receiving image information based upon determinations ofpupil size or orientation, or upon receiving electrical signalsindicative of particular pupil size or orientation. For example, if theuser's eyes are unable to distinguish between two depth planesassociated with two waveguides, then the controller 460 (which may be anembodiment of the local processing and data module 260) can beconfigured or programmed to cease providing image information to one ofthese waveguides. Advantageously, this may reduce the processing burdenon the system, thereby increasing the responsiveness of the system. Inembodiments in which the DOEs for a waveguide are switchable between theon and off states, the DOEs may be switched to the off state when thewaveguide does receive image information.

In some embodiments, it may be desirable to have an exit beam meet thecondition of having a diameter that is less than the diameter of the eyeof a viewer. However, meeting this condition may be challenging in viewof the variability in size of the viewer's pupils. In some embodiments,this condition is met over a wide range of pupil sizes by varying thesize of the exit beam in response to determinations of the size of theviewer's pupil. For example, as the pupil size decreases, the size ofthe exit beam may also decrease. In some embodiments, the exit beam sizemay be varied using a variable aperture.

The wearable system 400 can include an outward-facing imaging system 464(e.g., a digital camera) that images a portion of the world 470. Thisportion of the world 470 may be referred to as the field of view (FOV)of a world camera and the imaging system 464 is sometimes referred to asan FOV camera. The FOV of the world camera may or may not be the same asthe FOV of a viewer 210 which encompasses a portion of the world 470 theviewer 210 perceives at a given time. For example, in some situations,the FOV of the world camera may be larger than the viewer 210 of theviewer 210 of the wearable system 400. The entire region available forviewing or imaging by a viewer may be referred to as the field of regard(FOR). The FOR may include 4π steradians of solid angle surrounding thewearable system 400 because the wearer can move his body, head, or eyesto perceive substantially any direction in space. In other contexts, thewearer's movements may be more constricted, and accordingly the wearer'sFOR may subtend a smaller solid angle. Images obtained from theoutward-facing imaging system 464 can be used to track gestures made bythe user (e.g., hand or finger gestures), detect objects in the world470 in front of the user, and so forth.

The wearable system 400 can include an audio sensor 232, e.g., amicrophone, to capture ambient sound. As described above, in someembodiments, one or more other audio sensors can be positioned toprovide stereo sound reception useful to the determination of locationof a speech source. The audio sensor 232 can comprise a directionalmicrophone, as another example, which can also provide such usefuldirectional information as to where the audio source is located. Thewearable system 400 can use information from both the outward-facingimaging system 464 and the audio sensor 230 in locating a source ofspeech, or to determine an active speaker at a particular moment intime, etc. For example, the wearable system 400 can use the voicerecognition alone or in combination with a reflected image of thespeaker (e.g., as seen in a mirror) to determine the identity of thespeaker. As another example, the wearable system 400 can determine aposition of the speaker in an environment based on sound acquired fromdirectional microphones. The wearable system 400 can parse the soundcoming from the speaker's position with speech recognition algorithms todetermine the content of the speech and use voice recognition techniquesto determine the identity (e.g., name or other demographic information)of the speaker.

The wearable system 400 can also include an inward-facing imaging system466 (e.g., a digital camera), which observes the movements of the user,such as the eye movements and the facial movements. The inward-facingimaging system 466 may be used to capture images of the eye 410 todetermine the size and/or orientation of the pupil of the eye 304. Theinward-facing imaging system 466 can be used to obtain images for use indetermining the direction the user is looking (e.g., eye pose) or forbiometric identification of the user (e.g., via iris identification). Insome embodiments, at least one camera may be utilized for each eye, toseparately determine the pupil size or eye pose of each eyeindependently, thereby allowing the presentation of image information toeach eye to be dynamically tailored to that eye. In some otherembodiments, the pupil diameter or orientation of only a single eye 410(e.g., using only a single camera per pair of eyes) can be determinedand assumed to be similar for both eyes of the user. The images obtainedby the inward-facing imaging system 466 may be analyzed to determine theuser's eye pose or mood, which can be used by the wearable system 400 todecide which audio or visual content should be presented to the user.The wearable system 400 may also determine head pose (e.g., headposition or head orientation) using sensors such as IMUs,accelerometers, gyroscopes, etc.

The wearable system 400 can include a user input device 466 by which theuser can input commands to the controller 460 to interact with thewearable system 400. For example, the user input device 466 can includea trackpad, a touchscreen, a joystick, a multiple degree-of-freedom(DOF) controller, a capacitive sensing device, a game controller, akeyboard, a mouse, a directional pad (D-pad), a wand, a haptic device, atotem (e.g., functioning as a virtual user input device), and so forth.A multi-DOF controller can sense user input in some or all possibletranslations (e.g., left/right, forward/backward, or up/down) orrotations (e.g., yaw, pitch, or roll) of the controller. A multi-DOFcontroller which supports the translation movements may be referred toas a 3DOF while a multi-DOF controller which supports the translationsand rotations may be referred to as 6DOF. In some cases, the user mayuse a finger (e.g., a thumb) to press or swipe on a touch-sensitiveinput device to provide input to the wearable system 400 (e.g., toprovide user input to a user interface provided by the wearable system400). The user input device 466 may be held by the user's hand duringthe use of the wearable system 400. The user input device 466 can be inwired or wireless communication with the wearable system 400.

Other Components of the Wearable System

In many implementations, the wearable system may include othercomponents in addition or in alternative to the components of thewearable system described above. The wearable system may, for example,include one or more haptic devices or components. The haptic devices orcomponents may be operable to provide a tactile sensation to a user. Forexample, the haptic devices or components may provide a tactilesensation of pressure or texture when touching virtual content (e.g.,virtual objects, virtual tools, other virtual constructs). The tactilesensation may replicate a feel of a physical object which a virtualobject represents, or may replicate a feel of an imagined object orcharacter (e.g., a dragon) which the virtual content represents. In someimplementations, haptic devices or components may be worn by the user(e.g., a user wearable glove). In some implementations, haptic devicesor components may be held by the user.

The wearable system may, for example, include one or more physicalobjects which are manipulable by the user to allow input or interactionwith the wearable system. These physical objects may be referred toherein as totems. Some totems may take the form of inanimate objects,such as for example, a piece of metal or plastic, a wall, a surface oftable. In certain implementations, the totems may not actually have anyphysical input structures (e.g., keys, triggers, joystick, trackball,rocker switch). Instead, the totem may simply provide a physicalsurface, and the wearable system may render a user interface so as toappear to a user to be on one or more surfaces of the totem. Forexample, the wearable system may render an image of a computer keyboardand trackpad to appear to reside on one or more surfaces of a totem. Forexample, the wearable system may render a virtual computer keyboard andvirtual trackpad to appear on a surface of a thin rectangular plate ofaluminum which serves as a totem. The rectangular plate may not itselfhave any physical keys or trackpad or sensors. However, the wearablesystem may detect user manipulation or interaction or touches with therectangular plate as selections or inputs made via the virtual keyboardor virtual trackpad. The user input device 466 (shown in FIG. 4 ) may bean embodiment of a totem, which may include a trackpad, a touchpad, atrigger, a joystick, a trackball, a rocker or virtual switch, a mouse, akeyboard, a multi-degree-of-freedom controller, or another physicalinput device. A user may use the totem, alone or in combination withposes, to interact with the wearable system or other users.

Example Processes of User Interactions with a Wearable System

FIG. 5 is a process flow diagram of an example of a method 500 forinteracting with a virtual user interface. The method 500 may beperformed by the wearable system described herein. Embodiments of themethod 500 can be used by the wearable system to detect persons ordocuments in the FOV of the wearable system.

At block 510, the wearable system may identify a particular UI. The typeof UI may be predetermined by the user. The wearable system may identifythat a particular UI needs to be populated based on a user input (e.g.,gesture, visual data, audio data, sensory data, direct command, etc.).The UI can be specific to a security scenario where the wearer of thesystem is observing users who present documents to the wearer (e.g., ata travel checkpoint). At block 520, the wearable system may generatedata for the virtual UI. For example, data associated with the confines,general structure, shape of the UI etc., may be generated. In addition,the wearable system may determine map coordinates of the user's physicallocation so that the wearable system can display the UI in relation tothe user's physical location. For example, if the UI is body centric,the wearable system may determine the coordinates of the user's physicalstance, head pose, or eye pose such that a ring UI can be displayedaround the user or a planar UI can be displayed on a wall or in front ofthe user. In the security context described herein, the UI may bedisplayed as if the UI were surrounding the traveler who is presentingdocuments to the wearer of the system, so that the wearer can readilyview the UI while looking at the traveler and the traveler's documents.If the UI is hand centric, the map coordinates of the user's hands maybe determined. These map points may be derived through data receivedthrough the FOV cameras, sensory input, or any other type of collecteddata.

At block 530, the wearable system may send the data to the display fromthe cloud or the data may be sent from a local database to the displaycomponents. At block 540, the UI is displayed to the user based on thesent data. For example, a light field display can project the virtual UIinto one or both of the user's eyes. Once the virtual UI has beencreated, the wearable system may simply wait for a command from the userto generate more virtual content on the virtual UI at block 550. Forexample, the UI may be a body centric ring around the user's body or thebody of a person in the user's environment (e.g., a traveler). Thewearable system may then wait for the command (a gesture, a head or eyemovement, voice command, input from a user input device, etc.), and ifit is recognized (block 560), virtual content associated with thecommand may be displayed to the user (block 570).

Examples of Avatar Rendering in Mixed Reality

A wearable system may employ various mapping related techniques in orderto achieve high depth of field in the rendered light fields. In mappingout the virtual world, it is advantageous to know all the features andpoints in the real world to accurately portray virtual objects inrelation to the real world. To this end, FOV images captured from usersof the wearable system can be added to a world model by including newpictures that convey information about various points and features ofthe real world. For example, the wearable system can collect a set ofmap points (such as 2D points or 3D points) and find new map points torender a more accurate version of the world model. The world model of afirst user can be communicated (e.g., over a network such as a cloudnetwork) to a second user so that the second user can experience theworld surrounding the first user.

FIG. 6A is a block diagram of another example of a wearable system whichcan comprise an avatar processing and rendering system 690 in a mixedreality environment. The wearable system 600 may be part of the wearablesystem 200 shown in FIG. 2 . In this example, the wearable system 600can comprise a map 620, which may include at least a portion of the datain the map database 710 (shown in FIG. 7 ). The map may partly residelocally on the wearable system, and may partly reside at networkedstorage locations accessible by wired or wireless network (e.g., in acloud system). A pose process 610 may be executed on the wearablecomputing architecture (e.g., processing module 260 or controller 460)and utilize data from the map 620 to determine position and orientationof the wearable computing hardware or user. Pose data may be computedfrom data collected on the fly as the user is experiencing the systemand operating in the world. The data may comprise images, data fromsensors (such as inertial measurement units, which generally compriseaccelerometer and gyroscope components) and surface informationpertinent to objects in the real or virtual environment.

A sparse point representation may be the output of a simultaneouslocalization and mapping (e.g., SLAM or vSLAM, referring to aconfiguration wherein the input is images/visual only) process. Thesystem can be configured to not only find out where in the world thevarious components are, but what the world is made of. Pose may be abuilding block that achieves many goals, including populating the mapand using the data from the map.

In one embodiment, a sparse point position may not be completelyadequate on its own, and further information may be needed to produce amultifocal AR, VR, or MR experience. Dense representations, generallyreferring to depth map information, may be utilized to fill this gap atleast in part. Such information may be computed from a process referredto as Stereo 640, wherein depth information is determined using atechnique such as triangulation or time-of-flight sensing. Imageinformation and active patterns (such as infrared patterns created usingactive projectors), images acquired from image cameras, or handgestures/totem 650 may serve as input to the Stereo process 640. Asignificant amount of depth map information may be fused together, andsome of this may be summarized with a surface representation. Forexample, mathematically definable surfaces may be efficient (e.g.,relative to a large point cloud) and digestible inputs to otherprocessing devices like game engines. Thus, the output of the stereoprocess (e.g., a depth map) 640 may be combined in the fusion process630. Pose 610 may be an input to this fusion process 630 as well, andthe output of fusion 630 becomes an input to populating the map process620. Sub-surfaces may connect with each other, such as in topographicalmapping, to form larger surfaces, and the map becomes a large hybrid ofpoints and surfaces.

To resolve various aspects in a mixed reality process 660, variousinputs may be utilized. For example, in the embodiment depicted in FIG.6A, game parameters may be inputs to determine that the user of thesystem is playing a monster battling game with one or more monsters atvarious locations, monsters dying or running away under variousconditions (such as if the user shoots the monster), walls or otherobjects at various locations, and the like. The world map may includeinformation regarding the location of the objects or semanticinformation of the objects (e.g., classifications such as whether theobject is flat or round, horizontal or vertical, a table or a lamp,etc.) and the world map can be another valuable input to mixed reality.Pose relative to the world can become an input as well and can play akey role to almost any interactive system.

Controls or inputs from the user can be another input to the wearablesystem 600. As described herein, user inputs can include visual input,gestures, totems, audio input, sensory input, etc. In order to movearound or play a game, for example, the user may need to instruct thewearable system 600 regarding what he or she wants to do. Beyond justmoving oneself in space, there can be various forms of user controlsthat may be utilized. In one embodiment, a totem (e.g. a user inputdevice), or an object such as a toy gun may be held by the user andtracked by the system. The system preferably will be configured to knowthat the user is holding the item and understand what kind ofinteraction the user is having with the item (e.g., if the totem orobject is a gun, the system may be configured to understand location andorientation, as well as whether the user is clicking a trigger or othersensed button or element which may be equipped with a sensor, such as anIMU, which may assist in determining what is going on, even when suchactivity is not within the field of view of any of the cameras.)

Hand gesture tracking or recognition may also provide input information.The wearable system 600 may be configured to track and interpret handgestures for button presses, for gesturing left or right, stop, grab,hold, etc. For example, in one configuration, the user may want to flipthrough emails or a calendar in a non-gaming environment, or do a “fistbump” with another person or player. The wearable system 600 may beconfigured to leverage a minimum amount of hand gesture, which may ormay not be dynamic. For example, the gestures may be simple staticgestures like open hand for stop, thumbs up for ok, thumbs down for notok; or a hand flip right, or left, or up/down for directional commands.

Eye tracking can be another input (e.g., tracking where the user islooking to control the display technology to render at a specific depthor range). In one embodiment, vergence of the eyes may be determinedusing triangulation, and then using a vergence/accommodation modeldeveloped for that particular person, accommodation may be determined.Eye tracking can be performed by the eye camera(s) to determine eye gaze(e.g., direction or orientation of one or both eyes). Other techniquescan be used for eye tracking such as, e.g., measurement of electricalpotentials by electrodes placed near the eye(s) (e.g.,electrooculography).

Speech tracking can be another input and can be used alone or incombination with other inputs (e.g., totem tracking, eye tracking,gesture tracking, etc.). Speech tracking may include speech recognition,voice recognition, alone or in combination. The system 600 can includean audio sensor (e.g., a microphone) that receives an audio stream fromthe environment. The system 600 can incorporate voice recognitiontechnology to determine who is speaking (e.g., whether the speech isfrom the wearer of the ARD or another person or voice (e.g., a recordedvoice transmitted by a loudspeaker in the environment)) as well asspeech recognition technology to determine what is being said. The localdata & processing module 260 or the remote processing module 270 canprocess the audio data from the microphone (or audio data in anotherstream such as, e.g., a video stream being watched by the user) toidentify content of the speech by applying various speech recognitionalgorithms, such as, e.g., hidden Markov models, dynamic time warping(DTW)-based speech recognitions, neural networks, deep learningalgorithms such as deep feedforward and recurrent neural networks,end-to-end automatic speech recognitions, machine learning algorithms(described with reference to FIG. 7 ), or other algorithms that usesacoustic modeling or language modeling, etc.

The local data & processing module 260 or the remote processing module270 can also apply voice recognition algorithms which can identify theidentity of the speaker, such as whether the speaker is the user 210 ofthe wearable system 600 or another person with whom the user isconversing. Some example voice recognition algorithms can includefrequency estimation, hidden Markov models, Gaussian mixture models,pattern matching algorithms, neural networks, matrix representation,Vector Quantization, speaker diarisation, decision trees, and dynamictime warping (DTW) technique. Voice recognition techniques can alsoinclude anti-speaker techniques, such as cohort models, and worldmodels. Spectral features may be used in representing speakercharacteristics. The local data & processing module or the remote dataprocessing module 270 can use various machine learning algorithmsdescribed with reference to FIG. 7 to perform the voice recognition.

An implementation of a wearable system can use these user controls orinputs via a UI. UI elements (e.g., controls, popup windows, bubbles,data entry fields, etc.) can be used, for example, to dismiss a displayof information, e.g., graphics or semantic information of an object.

With regard to the camera systems, the example wearable system 600 shownin FIG. 6A can include three pairs of cameras: a relative wide FOV orpassive SLAM pair of cameras arranged to the sides of the user's face, adifferent pair of cameras oriented in front of the user to handle thestereo imaging process 640 and also to capture hand gestures andtotem/object tracking in front of the user's face. The FOV cameras andthe pair of cameras for the stereo process 640 may be a part of theoutward-facing imaging system 464 (shown in FIG. 4 ). The wearablesystem 600 can include eye tracking cameras (which may be a part of aninward-facing imaging system 462 shown in FIG. 4 ) oriented toward theeyes of the user in order to triangulate eye vectors and otherinformation. The wearable system 600 may also comprise one or moretextured light projectors (such as infrared (IR) projectors) to injecttexture into a scene.

The wearable system 600 can comprise an avatar processing and renderingsystem 690. The avatar processing and rendering system 690 can beconfigured to generate, update, animate, and render an avatar based oncontextual information. Some or all of the avatar processing andrendering system 690 can be implemented as part of the local processingand data module 260 or the remote processing module 262, 264 alone or incombination. In various embodiments, multiple avatar processing andrendering systems 690 (e.g., as implemented on different wearabledevices) can be used for rendering the virtual avatar 670. For example,a first user's wearable device may be used to determine the first user'sintent, while a second user's wearable device can determine an avatar'scharacteristics and render the avatar of the first user based on theintent received from the first user's wearable device. The first user'swearable device and the second user's wearable device (or other suchwearable devices) can communicate via a network, for example, as will bedescribed with reference to FIG. 9 .

FIG. 6B illustrates an example avatar processing and rendering system690. The example avatar processing and rendering system 690 can comprisea 3D model processing system 680, a contextual information analysissystem 688, an avatar autoscaler 692, an intent mapping system 694, ananatomy adjustment system 698, a stimuli response system 696, alone orin combination. The system 690 is intended to illustrate functionalitiesfor avatar processing and rendering and is not intended to be limiting.For example, in certain implementations, one or more of these systemsmay be part of another system. For example, portions of the contextualinformation analysis system 688 may be part of the avatar autoscaler692, intent mapping system 694, stimuli response system 696, or anatomyadjustment system 698, individually or in combination.

The contextual information analysis system 688 can be configured todetermine environment and object information based on one or more devicesensors described with reference to FIGS. 2 and 3 . For example, thecontextual information analysis system 688 can analyze environments andobjects (including physical or virtual objects) of a user's environmentor an environment in which the user's avatar is rendered, using imagesacquired by the outward-facing imaging system 464 of the user or theviewer of the user's avatar. The contextual information analysis system688 can analyze such images alone or in combination with a data acquiredfrom location data or world maps (e.g., maps 620, 710, 910) to determinethe location and layout of objects in the environments. The contextualinformation analysis system 688 can also access biological features ofthe user or human in general for animating the virtual avatar 670realistically. For example, the contextual information analysis system688 can generate a discomfort curve which can be applied to the avatarsuch that a portion of the user's avatar's body (e.g., the head) may notbe at an uncomfortable (or unrealistic) position with respect to theother portions of the user's body (e.g., the avatar's head is not turned270 degrees). In certain implementations, one or more object recognizers708 (shown in FIG. 7 ) may be implemented as part of the contextualinformation analysis system 688.

The avatar autoscaler 692, the intent mapping system 694, and thestimuli response system 696, and anatomy adjustment system 698 can beconfigured to determine the avatar's characteristics based on contextualinformation. Some example characteristics of the avatar can include thesize, appearance, position, orientation, movement, pose, expression,etc. The avatar autoscaler 692 can be configured to automatically scalethe avatar such that the user does not have to look at the avatar at anuncomfortable pose. For example, the avatar autoscaler 692 can increaseor decrease the size of the avatar to bring the avatar to the user's eyelevel such that the user does not need to look down at the avatar orlook up at the avatar respectively. The intent mapping system 694 candetermine an intent of a user's interaction and map the intent to anavatar (rather than the exact user interaction) based on the environmentthat the avatar is rendered in. For example, an intent of a first usermay be to communicate with a second user in a telepresence session (see,e.g., FIG. 9B). Typically, two people face each other whencommunicating. The intent mapping system 694 of the first user'swearable system can determine that such a face-to-face intent existsduring the telepresence session and can cause the first user's wearablesystem to render the second user's avatar to be facing the first user.If the second user were to physically turn around, instead of renderingthe second user's avatar in a turned position (which would cause theback of the second user's avatar to be rendered to the first user), thefirst user's intent mapping system 694 can continue to render the secondavatar's face to the first user, which is the inferred intent of thetelepresence session (e.g., face-to-face intent in this example).

The stimuli response system 696 can identify an object of interest inthe environment and determine an avatar's response to the object ofinterest. For example, the stimuli response system 696 can identify asound source in an avatar's environment and automatically turn theavatar to look at the sound source. The stimuli response system 696 canalso determine a threshold termination condition. For example, thestimuli response system 696 can cause the avatar to go back to itsoriginal pose after the sound source disappears or after a period oftime has elapsed.

The anatomy adjustment system 698 can be configured to adjust the user'spose based on biological features. For example, the anatomy adjustmentsystem 698 can be configured to adjust relative positions between theuser's head and the user's torso or between the user's upper body andlower body based on a discomfort curve.

The 3D model processing system 680 can be configured to animate andcause the display 220 to render a virtual avatar 670. The 3D modelprocessing system 680 can include a virtual character processing system682 and a movement processing system 684. The virtual characterprocessing system 682 can be configured to generate and update a 3Dmodel of a user (for creating and animating the virtual avatar). Themovement processing system 684 can be configured to animate the avatar,such as, e.g., by changing the avatar's pose, by moving the avatararound in a user's environment, or by animating the avatar's facialexpressions, etc. As will further be described herein, the virtualavatar can be animated using rigging techniques. In some embodiments, anavatar is represented in two parts: a surface representation (e.g., adeformable mesh) that is used to render the outward appearance of thevirtual avatar and a hierarchical set of interconnected joints (e.g., acore skeleton) for animating the mesh. In some implementations, thevirtual character processing system 682 can be configured to edit orgenerate surface representations, while the movement processing system684 can be used to animate the avatar by moving the avatar, deformingthe mesh, etc.

Examples of Mapping a User's Environment

FIG. 7 is a block diagram of an example of an MR environment 700. The MRenvironment 700 may be configured to receive input (e.g., visual input702 from the user's wearable system, stationary input 704 such as roomcameras, sensory input 706 from various sensors, gestures, totems, eyetracking, user input from the user input device 466 etc.) from one ormore user wearable systems (e.g., wearable system 200 or display system220) or stationary room systems (e.g., room cameras, etc.). The wearablesystems can use various sensors (e.g., accelerometers, gyroscopes,temperature sensors, movement sensors, depth sensors, GPS sensors,inward-facing imaging system, outward-facing imaging system, etc.) todetermine the location and various other attributes of the environmentof the user. This information may further be supplemented withinformation from stationary cameras in the room that may provide imagesor various cues from a different point of view. The image data acquiredby the cameras (such as the room cameras and/or the cameras of theoutward-facing imaging system) may be reduced to a set of mappingpoints.

One or more object recognizers 708 can crawl through the received data(e.g., the collection of points) and recognize or map points, tagimages, attach semantic information to objects with the help of a mapdatabase 710. The map database 710 may comprise various points collectedover time and their corresponding objects. The various devices and themap database can be connected to each other through a network (e.g.,LAN, WAN, etc.) to access the cloud.

Based on this information and collection of points in the map database,the object recognizers 708 a to 708 n may recognize objects in anenvironment. For example, the object recognizers can recognize faces,persons, windows, walls, user input devices, televisions, documents(e.g., travel tickets, driver's license, passport as described in thesecurity examples herein), other objects in the user's environment, etc.One or more object recognizers may be specialized for object withcertain characteristics. For example, the object recognizer 708 a may beused to recognizer faces, while another object recognizer may be usedrecognize documents.

The object recognitions may be performed using a variety of computervision techniques. For example, the wearable system can analyze theimages acquired by the outward-facing imaging system 464 (shown in FIG.4 ) to perform scene reconstruction, event detection, video tracking,object recognition (e.g., persons or documents), object pose estimation,facial recognition (e.g., from a person in the environment or an imageon a document), learning, indexing, motion estimation, or image analysis(e.g., identifying indicia within documents such as photos, signatures,identification information, travel information, etc.), and so forth. Oneor more computer vision algorithms may be used to perform these tasks.Non-limiting examples of computer vision algorithms include:Scale-invariant feature transform (SIFT), speeded up robust features(SURF), oriented FAST and rotated BRIEF (ORB), binary robust invariantscalable keypoints (BRISK), fast retina keypoint (FREAK), Viola-Jonesalgorithm, Eigenfaces approach, Lucas-Kanade algorithm, Horn-Schunkalgorithm, Mean-shift algorithm, visual simultaneous location andmapping (vSLAM) techniques, a sequential Bayesian estimator (e.g.,Kalman filter, extended Kalman filter, etc.), bundle adjustment,Adaptive thresholding (and other thresholding techniques), IterativeClosest Point (ICP), Semi Global Matching (SGM), Semi Global BlockMatching (SGBM), Feature Point Histograms, various machine learningalgorithms (such as e.g., support vector machine, k-nearest neighborsalgorithm, Naive Bayes, neural network (including convolutional or deepneural networks), or other supervised/unsupervised models, etc.), and soforth.

The object recognitions can additionally or alternatively be performedby a variety of machine learning algorithms. Once trained, the machinelearning algorithm can be stored by the HMD. Some examples of machinelearning algorithms can include supervised or non-supervised machinelearning algorithms, including regression algorithms (such as, forexample, Ordinary Least Squares Regression), instance-based algorithms(such as, for example, Learning Vector Quantization), decision treealgorithms (such as, for example, classification and regression trees),Bayesian algorithms (such as, for example, Naive Bayes), clusteringalgorithms (such as, for example, k-means clustering), association rulelearning algorithms (such as, for example, a-priori algorithms),artificial neural network algorithms (such as, for example, Perceptron),deep learning algorithms (such as, for example, Deep Boltzmann Machine,or deep neural network), dimensionality reduction algorithms (such as,for example, Principal Component Analysis), ensemble algorithms (suchas, for example, Stacked Generalization), and/or other machine learningalgorithms. In some embodiments, individual models can be customized forindividual data sets. For example, the wearable device can generate orstore a base model. The base model may be used as a starting point togenerate additional models specific to a data type (e.g., a particularuser in the telepresence session), a data set (e.g., a set of additionalimages obtained of the user in the telepresence session), conditionalsituations, or other variations. In some embodiments, the wearable HMDcan be configured to utilize a plurality of techniques to generatemodels for analysis of the aggregated data. Other techniques may includeusing pre-defined thresholds or data values.

Based on this information and collection of points in the map database,the object recognizers 708 a to 708 n may recognize objects andsupplement objects with semantic information to give life to theobjects. For example, if the object recognizer recognizes a set ofpoints to be a door, the system may attach some semantic information(e.g., the door has a hinge and has a 90 degree movement about thehinge). If the object recognizer recognizes a set of points to be amirror, the system may attach semantic information that the mirror has areflective surface that can reflect images of objects in the room. Thesemantic information can include affordances of the objects as describedherein. For example, the semantic information may include a normal ofthe object. The system can assign a vector whose direction indicates thenormal of the object. Over time the map database can grow as the system(which may reside locally or may be accessible through a wirelessnetwork) accumulates more data from the world. Once the objects arerecognized, the information may be transmitted to one or more wearablesystems. For example, the MR environment 700 may include informationabout a scene happening in California. The environment 700 may betransmitted to one or more users in New York. Based on data receivedfrom an FOV camera and other inputs, the object recognizers and othersoftware components can map the points collected from the variousimages, recognize objects etc., such that the scene may be accurately“passed over” to a second user, who may be in a different part of theworld. The environment 700 may also use a topological map forlocalization purposes.

FIG. 8 is a process flow diagram of an example of a method 800 ofrendering virtual content in relation to recognized objects. The method800 describes how a virtual scene may be presented to a user of thewearable system. The user may be geographically remote from the scene.For example, the user may be in New York, but may want to view a scenethat is presently going on in California, or may want to go on a walkwith a friend who resides in California.

At block 810, the wearable system may receive input from the user andother users regarding the environment of the user. This may be achievedthrough various input devices, and knowledge already possessed in themap database. The user's FOV camera, sensors, GPS, eye tracking, etc.,convey information to the system at block 810. The system may determinesparse points based on this information at block 820. The sparse pointsmay be used in determining pose data (e.g., head pose, eye pose, bodypose, or hand gestures) that can be used in displaying and understandingthe orientation and position of various objects in the user'ssurroundings. The object recognizers 708 a-708 n may crawl through thesecollected points and recognize one or more objects using a map databaseat block 830. This information may then be conveyed to the user'sindividual wearable system at block 840, and the desired virtual scenemay be accordingly displayed to the user at block 850. For example, thedesired virtual scene (e.g., user in CA) may be displayed at theappropriate orientation, position, etc., in relation to the variousobjects and other surroundings of the user in New York.

Example Communications among Multiple Wearable Systems

FIG. 9 schematically illustrates an overall system view depictingmultiple user devices interacting with each other. The computingenvironment 900 includes user devices 930 a, 930 b, 930 c. The userdevices 930 a, 930 b, and 930 c can communicate with each other througha network 990. The user devices 930 a-930 c can each include a networkinterface to communicate via the network 990 with a remote computingsystem 920 (which may also include a network interface 971). The network990 may be a LAN, WAN, peer-to-peer network, radio, Bluetooth, or anyother network. The computing environment 900 can also include one ormore remote computing systems 920. The remote computing system 920 mayinclude server computer systems that are clustered and located atdifferent geographic locations. The user devices 930 a, 930 b, and 930 cmay communicate with the remote computing system 920 via the network990.

The remote computing system 920 may include a remote data repository 980which can maintain information about a specific user's physical and/orvirtual worlds. Data storage 980 can store information related to users,users' environment (e.g., world maps of the user's environment), orconfigurations of avatars of the users. The remote data repository maybe an embodiment of the remote data repository 280 shown in FIG. 2 . Theremote computing system 920 may also include a remote processing module970. The remote processing module 970 may be an embodiment of the remoteprocessing module 270 shown in FIG. 2 . The remote processing module 970may include one or more processors which can communicate with the userdevices (930 a, 930 b, 930 c) and the remote data repository 980. Theprocessors can process information obtained from user devices and othersources. In some implementations, at least a portion of the processingor storage can be provided by the local processing and data module 260(as shown in FIG. 2 ). The remote computing system 920 may enable agiven user to share information about the specific user's own physicaland/or virtual worlds with another user.

The user device may be a wearable device (such as an HMD or an ARD), acomputer, a mobile device, or any other devices alone or in combination.For example, the user devices 930 b and 930 c may be an embodiment ofthe wearable system 200 shown in FIG. 2 (or the wearable system 400shown in FIG. 4 ) which can be configured to present AR/VR/MR content.

One or more of the user devices can be used with the user input device466 shown in FIG. 4 . A user device can obtain information about theuser and the user's environment (e.g., using the outward-facing imagingsystem 464 shown in FIG. 4 ). The user device and/or remote computingsystem 1220 can construct, update, and build a collection of images,points and other information using the information obtained from theuser devices. For example, the user device may process raw informationacquired and send the processed information to the remote computingsystem 1220 for further processing. The user device may also send theraw information to the remote computing system 1220 for processing. Theuser device may receive the processed information from the remotecomputing system 1220 and provide final processing before projecting tothe user. The user device may also process the information obtained andpass the processed information to other user devices. The user devicemay communicate with the remote data repository 1280 while processingacquired information. Multiple user devices and/or multiple servercomputer systems may participate in the construction and/or processingof acquired images.

The information on the physical worlds may be developed over time andmay be based on the information collected by different user devices.Models of virtual worlds may also be developed over time and be based onthe inputs of different users. Such information and models can sometimesbe referred to herein as a world map or a world model. As described withreference to FIGS. 6 and 7 , information acquired by the user devicesmay be used to construct a world map 910. The world map 910 may includeat least a portion of the map 620 described in FIG. 6A. Various objectrecognizers (e.g. 708 a, 708 b, 708 c . . . 708 n) may be used torecognize objects and tag images, as well as to attach semanticinformation to the objects. These object recognizers are also describedin FIG. 7 .

The remote data repository 980 can be used to store data and tofacilitate the construction of the world map 910. The user device canconstantly update information about the user's environment and receiveinformation about the world map 910. The world map 910 may be created bythe user or by someone else. As discussed herein, user devices (e.g. 930a, 930 b, 930 c) and remote computing system 920, alone or incombination, may construct and/or update the world map 910. For example,a user device may be in communication with the remote processing module970 and the remote data repository 980. The user device may acquireand/or process information about the user and the user's environment.The remote processing module 970 may be in communication with the remotedata repository 980 and user devices (e.g. 930 a, 930 b, 930 c) toprocess information about the user and the user's environment. Theremote computing system 920 can modify the information acquired by theuser devices (e.g. 930 a, 930 b, 930 c), such as, e.g., selectivelycropping a user's image, modifying the user's background, adding virtualobjects to the user's environment, annotating a user's speech withauxiliary information, etc. The remote computing system 920 can send theprocessed information to the same and/or different user devices.

Example Process for 3D Model Sharing

FIG. 10 illustrates an example process 1000 of sharing 3D assets usingthe system and methods described herein. At step 1001, the 3D sharinghost application can access a 3D model file from the host system. Thehost may do this either by creating a new 3D asset on the host system orobtaining a 3D asset from an external system. The 3D asset may originatein a 3D modeling application such as AutoCad or 3D Studio Max, and maybe of any file type. The host may utilize an asset import plug-in (e.g.ASSIMP) to enable graphical format interchange. The ASSIMP plugin maysupport up to 40 different file formats, for example, such as Filmbox(FBX) in .fbx format or Autodesk Max 3D modeling (3DS) in .3ds format.In some embodiments, a custom program may be written to enable fileformat interchange. Any other suitable file format conversion method maybe used to enable file reformatting for use within the host 3D sharingapplication.

At step 1003, the 3D sharing host application can decompose the 3D assetinto constituent parts. Examples of constituent parts may be geometricdata, material data, vertex tables, one or more textures, triangleindices, or any other data used to ultimately define a fullrepresentation of a 3D model.

At step 1005, the constituent parts can be sent to clients that are partof a network with the host. In some embodiments, there may be only oneclient. In some embodiments, there may be two or more clients. In apreferred embodiment, a local area network may be utilized utilizing aUDP protocol. The network may be a LAN, WAN, peer-to-peer network,radio, Bluetooth, ad hoc, or any other network. In some embodiments, thenetwork may be the network 990 as described above. The host may be aremote computing system that may include server computer systems thatare clustered and located at different geographic locations, asdescribed above in context of 920. The client may be the user devices930 a, 930 b, and/or 930 c as described above. In some embodiments,there may be one host. In some embodiments, there may be two or morehosts. In some embodiments, each device connected to the network may beboth a host and a client, in effect both sharing and receiving 3D assetsfrom other devices on the network.

At step 1007, the client can receive the constituent parts.

At step 1009, the client can recompose the 3D asset from the constituentparts received from the host. In some embodiments, the 3D asset can bestored in the client device's memory. In some embodiments, the 3D assetmay be saved for later use by the device.

At step 1011, the client can display the recomposed 3D asset. In someembodiments, the client may not display the recomposed 3D asset, or maydisplay the asset at a later time, for example upon user request. Insome embodiments, the client may display the recomposed 3D asset, butthe user may not be able to see the recomposed 3D asset because it isoutside of the user's FOV.

Example 3D Model Sharing System Configuration

FIG. 11 illustrates an example 3D model sharing system configuration1100 for sharing 3D assets using the system and methods describedherein. FIG. 11 depicts a system configuration 1100 comprising oneclient (which may be called “remote”) system 1116 and one server (whichmay be called “host”) system 1118. The client system and the host systemneed not be physically separate systems; for example, in someembodiments, the client and the server can exist on the same physicaldevice (e.g., as client and server threads concurrently executed by oneor more processors of the same computer system). In practice, the 3Dsharing system 1100 may generally require at least one host system 1118that may function as a source of decomposed 3D assets. The host system1118 may have authoritative control over the type and quantity of the 3Dmodels available within the 3D sharing application. The 3D sharingsystem 1100 may have any number (e.g. zero, one, ten, etc.) of clientsystems 1116 operably connected to the host system 1118 through acommunication network 1114. The communication network 1114 may be a LAN,for example utilizing a User Datagram Protocol (UDP). The host system1118 may comprise one or more processors operably coupled to at leastone display, and capable of receiving a communication from othercomputing systems. In some embodiments, the host system 1118 may be a PCdesktop computer attached to a computer monitor with a networkinterface. In some embodiments, the host system 1118 may be system 200,930 a, 930 b, or 930 c as described above. The user devices 200, 930 a,930 b, and 930 c can communicate with each other and other computingsystems through a communication network 1114. The user devices 930 a-930c can each include a network interface to communicate via the network1114 with remote computing systems, such as 920 (which may also includea network interface 971). The client system 1116 may comprise one ormore processors operably coupled to at least one display, and capable ofreceiving a communication from other computing systems. In someembodiments, the client system 1116 may be a PC desktop computerattached to a computer monitor with a network interface. In someembodiments, the client system 1118 may be system 200, 930 a, 930 b, or930 c as described above. The user devices 200, 930 a, 930 b, and 930 ccan communicate with each other and other computing systems through acommunication network 1114. The user devices 930 a-930 c can eachinclude a network interface to communicate via the network 1114 withremote computing systems, such as 920 (which may also include a networkinterface 971).

The host system 1118 may have the 3D sharing host application installed.The 3D sharing host application may comprise three main modules: anassets module 1108, a decomposition module 1110, and a transmissionmodule 1112. The assets module 1108 may comprise one or more full 3Dmodels. The decomposition module 1110 may function to break a full 3Dmodel into constituent parts, optionally compress the constituent parts,and place the constituent parts into one or more arrays that are readyto send to a different system. The arrays may be stored in one or morelibraries that may contain one or more decomposed 3D models. Thetransmission module 1112 may function to break the constituent partsfrom the decomposition module 1110 into transferrable pieces of data andmay manage the transmission protocols and processes. The transmissionmodule 1112 may send the transferrable pieces of data to other systemsconnected on the communications network 1114.

The client, or remote, system 1116 may have a 3D sharing clientapplication installed. The 3D sharing client application may comprisethree main modules: a transmission module 1106, a recomposition module1104, an assets module 1102. The transmission module 1106 may functionto receive one or more decomposed 3D models and may manage thetransmission and networking processes for the client system 1116. Thetransmission module 1106 may communicate with the 3D model sharing hostapplication through messaging protocols. These protocols may be custombuilt protocols, or may utilize commonly used protocols. Therecomposition module 1104 may function to reassemble the receiveddecomposed 3D models from other systems on the network. The assetsmodule 1102 may comprise one or more full 3D models.

Example Process for 3D Model Sharing

FIG. 12 illustrates an example 3D model sharing process 1200 between aserver and client using the system and methods described herein. Theprocess 1200 may begin at step 1202, when a server system, such asserver system 1118, opens a 3D model sharing host application.

At step 1204, the server may load one or more full 3D model files intothe 3D model sharing host applications. The full 3D model file mayoriginate on the host system 1118 from a different 3D modeling softwaredownloaded on the host system 1118, such as AutoCad, for example. Insome embodiments, the 3D model file may be imported from an outsidesource, such as an external hard drive, a remote computing system (ex:920 from FIG. 9 above), or from cloud storage. In some embodiments, the3D model file may originate from a 3D scanner or a similar device. The3D model file may be imported using an ASSIMP, or equivalent, plug-in.The plug-in may convert the 3D model file from one format to a differentfile format that is compatible with the 3D model sharing hostapplication. The plug-in may only be utilized by the host system, suchas host system 1118 for example, because the file formatting maydisappear at the server during the decomposition process, as describedin more detail below.

At step 1206, the server system may decompose the 3D model into itsconstituent parts. Decomposing the 3D model can comprise identifyingconstituent parts within the 3D model. Decomposing the 3D model can benon-destructive, in that it does not result in a change to the 3D modelitself (e.g., a loss of information in the 3D model), or a change in howthe 3D model is rendered for viewing. Decomposing the 3D model cancomprise applying data-level changes to the 3D model and/or itsconstituent parts (e.g., applying data formatting changes, orcompressing constituent parts to facilitate sharing). The decompositionprocess may occur in decomposition module 1110, as described above.Constituent parts may be sub-sets of the data required to fully define a3D model. The constituent parts sub-sets may be defined by pre-existingbundles of data comprising the 3D model, such as vertex tables,geometric data, material data, etc. In some embodiments, the constituentparts sub-sets may be defined in other ways, such as by the 3D modelsharing application programmer.

At step 1214, a client system may open the 3D model sharing clientapplication. This may initiate a series of steps related to networkingprotocols, communications, and discovery.

At step 1216, the 3D model sharing client application may discover a 3Dmodel sharing host, if a host exists, via a network. The network may bea LAN (e.g. UDP). In some embodiments, the local network is private. Insome embodiments, a cloud service is not used.

At step 1208, the server may receive a connection request from theclient, and the server may accept the connection request.

At step 1210, the server may send a client info request back to theclient. The client info request may contain a list of everything in thelibrary of the 3D model sharing host application. The client inforequest may alternatively be called an asset inventory, or a table ofcontents.

At step 1218, the client may reply back to the server with dataspecifying which items are missing from the 3D model sharing clientapplication library, when compared to the 3D model sharing serverapplication library.

At step 1212, the server may respond by sending the client the itemsmissing from the 3D model sharing client application. In someembodiments, the client may have recently opened the 3D model sharingapplication, and thus can start with zero items in its library. In thissituation, the host may send its entire library to the client. In someembodiments, the libraries may be placed in memory, and may thus beerased when the 3D model sharing application is closed. In someembodiments, the libraries may be saved to disk, such as the clientsystem hard drive, so the 3D models may be accessed at a later date. Insome embodiments, the 3D model data that is sent to the client is a datastructure without a file format. At the end of this step, the 3D modelsharing host library may match the 3D model sharing client library. Inother words, both the host system and the client system may now containthe same constituent parts for the same set of 3D models.

In some embodiments, a client may update items that are outdated whencompared to items stored in a server. For example, the client inforequest may include version numbers associated with individual assets inan asset inventory. One or more assets in the server's asset inventorycan correspond to assets stored with the client. The client can identifyitems stored in a client asset library that have lower version numbersthan a corresponding item in the server's asset inventory. The clientcan send data to the server identifying items that require updating, andthe server can respond by sending the client the corresponding updateditems.

At step 1220, the client may recompose one or more 3D models from thenew items sent from the server. In some embodiments, the client may nowhave the same full 3D models as the server.

At step 1222, the user of the client system may now view the 3D modelsshared by the host system.

Example 3D Model Sharing System Configuration

FIG. 13 illustrates an example 3D model sharing system configuration1300 for sharing 3D assets using the system and methods describedherein.

The 3D model sharing system configuration 1300 may comprise a hostsystem 1306 and a client system 1304 operably coupled by a communicationlink 1302, such as a network. The client system 1304 may have the 3Dmodel sharing client application open and running, and thus be operatingin a runtime environment. The host system 1306 may have the 3D modelsharing host application open and running, and thus be operating in aruntime environment. Although FIG. 13 depicts one server system and onehost system, this is not meant to be limiting in scope, and is done forease of illustration. In practice, the 3D model sharing systemconfiguration 1300 may have only one host system 1306, more than onehost system 1306, one client system 1304, more than one client system1304, and/or one or more systems that function as both a host (e.g.sends 3D model data to other systems via communication 1302) and client(e.g. receives 3D model data from other systems via communication 1302).Further, in some embodiments, a single system can comprise both the hostsystem and the client system. In some embodiments, host system 1306 maybe host system 1118. In some embodiments, client system 1304 may beclient system 1116. The host system 1306 may comprise one or moreprocessors operably coupled to at least one display, for example a PCdesktop computer attached to a computer monitor. In some embodiments,the host system 1306 may be system 200, 930 a, 930 b, or 930 c asdescribed above. The client system 1304 may comprise one or moreprocessors operably coupled to at least one display, for example a PCdesktop computer attached to a computer monitor. In some embodiments,the client system 1304 may be system 200, 930 a, 930 b, or 930 c asdescribed above.

Host system 1306 may comprise a server module 1316. The server module1316 may manage functions and processes specific to being a server, suchas transmission and networking protocols. The server module 1316 mayreceive and process connection requests, like steps 1208, 1210, and/or1212 in process 1200. In some embodiments, the server module 1316 maycomprise transmission module 1112.

Host system 1306 may comprise a host load module 1320. The load modulemay contain functions and processes that enable the 3D model sharinghost application to import or load a 3D asset into the 3D model sharinghost application or onto the host system 1306. The load module 1320 maycomprise an ASSIMP plug-in. In some embodiments, step 1204 from process1200 may occur within the load module 1320. In some embodiments, step1001 from process 1000 may occur within the load module 1320. In someembodiments, the assets module 1108 may comprise the load module 1320.

Host system 1306 may comprise a host library manager module 1318. Thelibrary manager module 1318 may contain and manage the data andprocesses corresponding to 3D assets that have already been loaded intothe 3D asset sharing host application. The library manager module 1318may contain and manage the data and processes corresponding to the 3Dasset sharing host application's libraries. This may include thedecomposition process, maintaining the library table of contents, etc.In some embodiments, step 1206 from process 1200 may occur in thelibrary manager module 1318. In some embodiments, step 1003 from process1000 may occur in the library manager module 1318. In some embodiments,the library manager module 1318 may comprise the decomposition module1110 from system 1100.

Host system 1306 may comprise host world module 1322. The host worldmodule 1322 may manage the full 3D assets, the 3D virtual world, meshingof the real world, etc. and the integration between these. In someembodiments, the host world module 1322 may be considered a scenegraph,or the functional equivalent of a scenegraph. In some embodiments, theworld can draw from items that come out of the world library (i.e. fullyrecomposed 3D model) and create world objects out of them. In someembodiments, everything in the 3D virtual world can be represented as aworld object. The world object may be part of a world “class”. In someembodiments, the host world module 1322 contains the same data as theclient world module 1312, by step 1222 of process 1200 or by step 1011of process 1000 (i.e. after the 3D models have been shared andrecomposed locally at the client). The output from the host world module1322 may feed into the render path module 1324.

Host system 1306 may comprise host render path module 1324. The renderpath module 1324 may accept as input data from the host world module1322, put it through the host system 1306 render pipeline, and maydisplay one or more 3D models on a display of the host system 1306.

Client system 1304 may comprise a client module 1308. The client module1304 may manage functions and processes specific to being a client, suchas transmission and networking and/or server/client protocols. Theclient module 1308 may manage the server discovery process and otherconnection protocols. In some embodiments, this may include steps 1216and/or 1218 in process 1200. In some embodiments, the client module 1308may comprise transmission module 1106.

Client system 1304 may comprise a client library manager module 1310.The library manager module 1310 may contain and manage the data andprocesses corresponding to 3D asset constituent parts that have beenreceived from the 3D asset sharing host application. The library managermodule 1310 may contain and manage the data and processes correspondingto the 3D asset sharing client application's libraries. This may includethe recomposition process. In some embodiments, step 1220 from process1200 may occur in the library manager module 1310. In some embodiments,step 1009 from process 1000 may occur in the library manager module1310. In some embodiments, the library manager module 1310 may comprisethe recomposition module 1104 from system 1100. In some embodiments, theclient libraries contain the same data as the host libraries, by step1222 of process 1200 or by step 1011 of process 1000 (i.e. after the 3Dmodels have been shared and recomposed locally at the client).

Client system 1304 may comprise client world module 1312. The clientworld module 1312 may manage the fully recomposed 3D assets, the 3Dvirtual world, meshing of the real world, etc. and the integrationbetween these. In some embodiments, the client world module 1312 may beconsidered a scenegraph, or the functional equivalent of a scenegraph.In some embodiments, the world draws from items that come out of a worldlibrary (i.e. fully recomposed 3D model) and creates world objects outof them. In some embodiments, everything in the 3D virtual world isrepresented as a world object. The world object may be part of a world“class”. In some embodiments, the client world module 1312 can containthe same data as the host world module 1322, by step 1222 of process1200 or by step 1011 of process 1000 (i.e. after the 3D models have beenshared and recomposed locally at the client). The output from the clientworld module 1312 may feed into the render path module 1314.

Client system 1304 may comprise client render path module 1314. Therender path module 1314 may accept as input data from the client worldmodule 1312, put it through the client system 1304 render pipeline, andmay display one or more virtual 3D models on a display of the clientsystem 1304.

The example 3D model sharing system configuration 1300 can enable two ormore devices to share full 3D models during runtime. This can have theadvantage of adding new assets to an application on a computing systemfaster than traditional systems where the updates may need to wait untilthe application is closed.

Example Process for Decomposing a 3D Model

FIG. 14 illustrates an example process 1400 for decomposing a full 3Dmodel using the system and methods described herein. In someembodiments, the decomposition process 1400 may occur in host librarymanager module 1318. In some embodiments, the decomposition process 1400may occur in decomposition module 1110. In some embodiments, thedecomposition process 1400 may be step 1003 in process 1000. In someembodiments, the decomposition process 1400 may be step 1206 in process1200.

The process 1400 may start with a full 3D model that can be loadedwithin the 3D model sharing host application. At step 1402, the 3D modelsharing host application may copy one constituent part of a 3D model andplace the data representing the constituent part into an array. Someexample constituent parts may be one or more vertex positions, vertextables, geometric data, material data, textures, triangle index tables,etc. A constituent part may be a portion of the data required torepresent a full 3D model.

At step 1404, the constituent part may optionally be compressed.Standard compression techniques may be used.

At step 1406, the constituent part may be packaged.

At step 1408, the package may be stored in one of the 3D model sharinghost application's libraries. The 3D model sharing host application mayhave one or more libraries used to store and organize the packages. Insome embodiments, the library may comprise three libraries—a meshlibrary, a material library, and a texture library, as shown in FIG.16.Alternate library configurations may be used. The package may be storedin the corresponding library (e.g. textures may be stored in the texturelibrary).

At step 1410, the process may check the 3D model to see if there areadditional constituent parts that need to be added to the library. Ifthere are more constituent parts that have not been added to thelibrary, the process can start over again at step 1402 until allconstituent parts are added to the library. When all constituent partshave been added, the library may contain a complete record of the full3D model, as in step 1412.

In some embodiments, a 3D model can be decomposed into constituent partspre-defined by a 3D model application. For example, a 3D modelapplication (e.g., a computer-aided design program) can define a 3Dmodel in terms of different data elements (e.g., a mesh and a meshrenderer), each of which can comprise one or more defined constituentparts (e.g., a triangle array or a texture). Decomposing a 3D model inthat case can constitute applying an operation (e.g., data formatting)to each constituent part and storing it in memory. In some embodiments,methods and systems described herein can be used to decompose a 3D modelinto one or more subgroups defined by a user. For example, for a 3Dmodel of a car, a user can designate a collection ofvertices/parts/structures as the “engine,” a collection ofvertices/parts/structures as the “chassis,” a collection ofvertices/parts/structures as the “drivetrain,” and a collection ofvertices/parts/structures as the “wheels.” Each subgroup can then beoptionally compressed and sent to a device separately. In someembodiments, a 3D model can be decomposed into algorithmically definedsubgroups. For example, an algorithm can have been developed (e.g.,manually, semi-automatically, and/or automatically, for example, throughmachine learning) to identify a collection of vertices/parts/structuresas an “engine.” An algorithm can first identify the whole 3D model(e.g., first identifying the model as a car) before attempting toclassify a collection of vertices/parts/structures from a limited poolbased on the initial identification. An algorithm can also directlyattempt to classify a collection of vertices/parts/structures.

Example of a Full 3D Model

FIG. 15 illustrates an example full 3D model 1500 using the system andmethods described herein. In some embodiments, a full 3D model may be acomplete set of data describing a 3D renderable digital object. In someembodiments, a full 3D model may be a set of data representing a digitalor virtual object that is capable of display when added to a compatiblerender pipeline.

At 3D model component 1502, the 3D model can be assigned a name. Thename may be assigned by the user, the import process, and/or may be theoriginal file name used in the 3D model creation software, such as thefile name given to a 3D model in AutoCad. The 3D model may have two maintypes of data: mesh data 1504 and mesh renderer data 1506. The mesh datamay be broken down further into subsets of data, or constituent parts,such as constituent part A 1508, constituent part B 1510, constituentpart 1512, and constituent part 1514. The material data 1516 may bebroken down further into subsets of data, or constituent parts, such asmaterial properties 1522, Texture 1 1518, and Texture 2 1522. The exactlayout, categorization, and sub-grouping of 3D model data may vary fromthat shown in FIG. 15 . In some embodiments, there may be any number ofconstituent parts under mesh 1504, such as two, three, or tenconstituent parts. In some embodiments, there may be multiple subsets ofproperties 1522 under material data 1516, instead of just one. In someembodiments, there may be more or fewer Texture constituent parts 1518,1520, than is shown in FIG. 15 . Any suitable number and categorizationmay be used to break down a full 3D model into constituent parts.

In some embodiments, exemplary full 3D model 1500 can be decomposedusing exemplary decomposition process 1400. Full 3D model 1500 cancomprise one or more constituent parts (e.g., constituent part A 1508,constituent part B 1510, constituent part C 1512, constituent part D1514, Texture 1 1518, Texture 2 1520, and/or material properties 1522).One constituent part of full 3D model 1500 (e.g., Texture 1) can becopied at step 1402 of decomposition process 1400. The constituent partcan be optionally compressed at step 1404 of decomposition process 1400.The constituent part can be packaged at step 1406 of decompositionprocess 1400. The constituent part can be stored at step 1408 ofdecomposition process 1400 (e.g., in exemplary set of libraries 1600).Steps 1402, 1404, 1406, and 1408 can be repeated for each constituentpart of full 3D model 1500 until all constituent parts of full 3D model1500 have been stored (e.g., in set of libraries 1600).

Example of Libraries in a 3D Model Sharing Application

FIG. 16 illustrates an example set of libraries 1600 utilized in a 3Dmodel sharing application to store constituent parts of a 3D model usingthe system and methods described herein. The set of libraries 1600 maycomprise one or more libraries in order to store and organize the 3Dmodel constituent parts. In some embodiments, there may be threelibraries: a mesh library 1602, a material library 1604, and a texturelibrary 1606. The mesh library 1602 may contain a set of constituentparts, in the form of arrays, that describe the 3D model mesh. Examplesarrays that may be stored in the mesh library are a vertex buffer array1608, a normal buffer array 1610, a UV buffer array 1612, and/or atriangle array 1614. The mesh library may optionally also contain one ormore string values, such as Material “Name” 1616, Bitmap Texture 1“Name” 1618, and Bitmap Texture 2 “Name” 1620. These string values 1616,1618, 1620 may reference the other libraries, such as material library1604 and texture library 1606. In some embodiments, a full 3D model maybe represented when all of the data is pulled or referenced from themesh library 1602. In some embodiments, a full 3D model for “Name” maybe represented when all of the data is pulled or referenced from all ofthe libraries 1602, 1604, 1606 corresponding to a record, or single 3Dmodel, “Name”.

Material library 1604 may contain data representing properties of thematerial for a 3D model and the properties associated values. In someembodiments, texture library 1606 may contain one or more textures 1622,1624. In some embodiments, texture library 1606 may contain one or morebitmaps. In some embodiments, texture library 1606 may contain one ormore UV maps, UV coordinates, or other UV space data.

All of the data contained in libraries 1600 of the 3D sharingapplication may be combined to represent a full, renderable 3D model.

Example Process for Recomposing a 3D Model

FIG. 17 illustrates an example process 1700 for recomposing a full 3Dmodel from its constituent parts using the system and methods describedherein. In some embodiments, the process may be step 1220 in process1200. In some embodiments, the process 1700 may be step 1009 fromprocess 1000. In some embodiments, the process 1700 may occur in therecomposition module 1104. In some embodiments, the process 1700 mayoccur in library manager module 1310.

The process 1700 may begin at step 1702 by creating an empty object,sometimes called an empty game object. In some embodiments, the emptygame object may comprise an empty volume in space with a referencecoordinate system. In some embodiments, the empty object is essentiallya transform that exists in space—a position and orientation. In someembodiments, the empty object is an empty prism. In some embodiments,the empty object may correspond to 1502 in FIG. 15 .

After the empty object is created 1702, a new mesh 1704 and a new meshrenderer 1708 may be added to the empty game object. In someembodiments, adding a new mesh 1704 may correspond to mesh 1504 in FIG.15 and adding a new mesh renderer 1708 may correspond to mesh renderer1506 in FIG. 15 .

The process 1700 may proceed to steps 1706 and 1710, where the emptymesh buffers can be filled with the 3D model constituent parts thatcorrespond to mesh data and the empty mesh renderer buffers can befilled with the 3D model constituent parts that correspond to meshrender data. In some embodiments, steps 1706 and/or 1710 may un-compresscompressed data. In some embodiments, the mesh data may correspond toconstituent parts 1508, 1510, 1512, 1514 from FIG. 15 . In someembodiments, the mesh render data may correspond to constituent parts1518, 1520, 1522. The buffers may be filled with data from the 3D modelapplication's libraries that correspond to a single 3D model. In someembodiments, the empty mesh buffers may be filled 1706 with data frommesh library 1602. In some embodiments, the empty mesh render buffersmay be filled 1710 with data from material library 1604 and texturelibrary 1606. The process 1700 can be complete when all of the datacorresponding to a single 3D model (i.e. a single record within the 3Dmodel sharing application's library) has been added to the empty object.

FIG. 18 illustrates an example full 3D model 1800 using the system andmethods described herein. In some embodiments, the full 3D model 1800may be the output of process 1700. In some embodiments, the full 3Dmodel 1800 can be a more detailed version of example full 3D model 1500,where generic constituent component A can be vertex buffer 1808, genericconstituent component B can be normal buffer 1810, generic constituentcomponent C can be UV buffer 1812, and generic constituent component Dcan be triangle indices 1814. In some embodiments, triangle indices 1814may need to be filled in last.

In some embodiments, full 3D model 1800 may be the input to process1400, from which the constituent parts are copied. In some embodiments,full 3D model 1800 may be stored in world module 1322 and/or 1312 as aworld object. In some embodiments, full 3D model 1800 may be the 3Dmodel loaded into the 3D model sharing server application in step 1204of process 1200 and/or 1001 of process 1000. In some embodiments, full3D model 1800 may be the output from step 1220 of process 1200 and/orstep 1009 of process 1000. In some embodiments, full 3D model 1800 maybe an asset stored in assets module 1102 and/or 1108. The Object “Name”1802 may match the original file name that was loaded into the 3D modelsharing application, car.fbx for example, even though the 3D model is nolonger in FBX file format.

Other Examples of 3D Model Sharing

In some embodiments, a 3D model can originate from an MR device (e.g.,wearable system 200). An MR device can capture information about aphysical object, generate a 3D model of the physical object, decomposethe 3D model into constituent parts, send constituent parts to a clientdevice, and the client device can generate a 3D model based onconstituent parts. For example, object recognizers 708 can receive inputfrom outward facing imaging system 464, lidar sensors, depth sensors,RGB cameras, infrared cameras, time-of-flight cameras, and/or othersensors on an MR device. This input can be used in computer visionalgorithms and/or machine-learning trained algorithms to generate a 3Dmodel based on the physical object. An algorithm can generate a 3D modelby first identifying a physical object and then generating a 3D modelbased on a library of 3D models associated with the physical object. Insome embodiments, a 3D model can be generated from a physical object byassembling subgroups based on pre-defined subgroups in a library of 3Dmodels. In some embodiments, a 3D model can be decomposed into subgroupsdefined by a user. In some embodiments, a 3D model can be decomposedinto algorithmically defined subgroups. For example, an algorithm canhave been developed (e.g., manually, semi-automatically, and/orautomatically, for example, through machine learning) to identify acollection of vertices/parts/structures as a subgroup.

In some embodiments, an algorithm can also generate a 3D model directlyfrom a physical object. For example, an algorithm can identify surfacesand vertices of a physical object and generate a 3D model of the wholephysical object based on observed characteristics of the physicalobject. In some embodiments, an algorithm can identify localizedsurfaces and vertices of a physical object and generate a 3D model of alocalized portion and/or define a subgroup of the physical object basedon a library of subgroups. An algorithm can store data regarding theassociation of the subgroups to form a complete 3D model based on theconstituent parts.

The 3D model sharing systems and methods described in the presentdisclosure can provide a specific solution to the technical problem ofhow to move a renderable 3D model from one computing system to adifferent computing system during runtime. This can present animprovement over existing technology which only adds new renderable 3Dmodels to an application offline.

Although 3D models are discussed herein, it is also contemplated thatsystems and methods described in this application can apply to otherdigital assets other than 3D models. For example, 2D models can bedecomposed, transferred, and reconstructed using systems and methodsdescribed herein (e.g., into outlines and textures). In someembodiments, an animation can be decomposed, transferred, andreconstructed using systems and methods described herein (e.g., into aseries of meshes, which can each be decomposed further as describedherein). In some embodiments, 3D models can be flattened into 2D models(e.g., to display on a 2D screen or to create a 2D map).

Other Considerations

Each of the processes, methods, and algorithms described herein and/ordepicted in the attached figures may be embodied in, and fully orpartially automated by, code modules executed by one or more physicalcomputing systems, hardware computer processors, application-specificcircuitry, and/or electronic hardware configured to execute specific andparticular computer instructions. For example, computing systems caninclude general purpose computers (e.g., servers) programmed withspecific computer instructions or special purpose computers, specialpurpose circuitry, and so forth. A code module may be compiled andlinked into an executable program, installed in a dynamic link library,or may be written in an interpreted programming language. In someimplementations, particular operations and methods may be performed bycircuitry that is specific to a given function.

Further, certain implementations of the functionality of the presentdisclosure are sufficiently mathematically, computationally, ortechnically complex that application-specific hardware or one or morephysical computing devices (utilizing appropriate specialized executableinstructions) may be necessary to perform the functionality, forexample, due to the volume or complexity of the calculations involved orto provide results substantially in real-time. For example, animationsor video may include many frames, with each frame having millions ofpixels, and specifically programmed computer hardware is necessary toprocess the video data to provide a desired image processing task orapplication in a commercially reasonable amount of time.

Code modules or any type of data may be stored on any type ofnon-transitory computer-readable medium, such as physical computerstorage including hard drives, solid state memory, random access memory(RAM), read only memory (ROM), optical disc, volatile or non-volatilestorage, combinations of the same and/or the like. The methods andmodules (or data) may also be transmitted as generated data signals(e.g., as part of a carrier wave or other analog or digital propagatedsignal) on a variety of computer-readable transmission mediums,including wireless-based and wired/cable-based mediums, and may take avariety of forms (e.g., as part of a single or multiplexed analogsignal, or as multiple discrete digital packets or frames). The resultsof the disclosed processes or process steps may be stored, persistentlyor otherwise, in any type of non-transitory, tangible computer storageor may be communicated via a computer-readable transmission medium.

Any processes, blocks, states, steps, or functionalities in flowdiagrams described herein and/or depicted in the attached figures shouldbe understood as potentially representing code modules, segments, orportions of code which include one or more executable instructions forimplementing specific functions (e.g., logical or arithmetical) or stepsin the process. The various processes, blocks, states, steps, orfunctionalities can be combined, rearranged, added to, deleted from,modified, or otherwise changed from the illustrative examples providedherein. In some embodiments, additional or different computing systemsor code modules may perform some or all of the functionalities describedherein. The methods and processes described herein are also not limitedto any particular sequence, and the blocks, steps, or states relatingthereto can be performed in other sequences that are appropriate, forexample, in serial, in parallel, or in some other manner. Tasks orevents may be added to or removed from the disclosed exampleembodiments. Moreover, the separation of various system components inthe implementations described herein is for illustrative purposes andshould not be understood as requiring such separation in allimplementations. It should be understood that the described programcomponents, methods, and systems can generally be integrated together ina single computer product or packaged into multiple computer products.Many implementation variations are possible.

The processes, methods, and systems may be implemented in a network (ordistributed) computing environment. Network environments includeenterprise-wide computer networks, intranets, local area networks (LAN),wide area networks (WAN), personal area networks (PAN), cloud computingnetworks, crowd-sourced computing networks, the Internet, and the WorldWide Web. The network may be a wired or a wireless network or any othertype of communication network.

The systems and methods of the disclosure each have several innovativeaspects, no single one of which is solely responsible or required forthe desirable attributes disclosed herein. The various features andprocesses described above may be used independently of one another, ormay be combined in various ways. All possible combinations andsubcombinations are intended to fall within the scope of thisdisclosure. Various modifications to the implementations described inthis disclosure may be readily apparent to those skilled in the art, andthe generic principles defined herein may be applied to otherimplementations without departing from the spirit or scope of thisdisclosure. Thus, the claims are not intended to be limited to theimplementations shown herein, but are to be accorded the widest scopeconsistent with this disclosure, the principles and the novel featuresdisclosed herein.

Certain features that are described in this specification in the contextof separate implementations also can be implemented in combination in asingle implementation. Conversely, various features that are describedin the context of a single implementation also can be implemented inmultiple implementations separately or in any suitable subcombination.Moreover, although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination. No single feature orgroup of features is necessary or indispensable to each and everyembodiment.

Conditional language used herein, such as, among others, “can,” “could,”“might,” “may,” “e.g.,” and the like, unless specifically statedotherwise, or otherwise understood within the context as used, isgenerally intended to convey that certain embodiments include, whileother embodiments do not include, certain features, elements and/orsteps. Thus, such conditional language is not generally intended toimply that features, elements and/or steps are in any way required forone or more embodiments or that one or more embodiments necessarilyinclude logic for deciding, with or without author input or prompting,whether these features, elements and/or steps are included or are to beperformed in any particular embodiment. The terms “comprising,”“including,” “having,” and the like are synonymous and are usedinclusively, in an open-ended fashion, and do not exclude additionalelements, features, acts, operations, and so forth. Also, the term “or”is used in its inclusive sense (and not in its exclusive sense) so thatwhen used, for example, to connect a list of elements, the term “or”means one, some, or all of the elements in the list. In addition, thearticles “a,” “an,” and “the” as used in this application and theappended claims are to be construed to mean “one or more” or “at leastone” unless specified otherwise.

As used herein, a phrase referring to “at least one of” a list of itemsrefers to any combination of those items, including single members. Asan example, “at least one of: A, B, or C” is intended to cover: A, B, C,A and B, A and C, B and C, and A, B, and C. Conjunctive language such asthe phrase “at least one of X, Y and Z,” unless specifically statedotherwise, is otherwise understood with the context as used in generalto convey that an item, term, etc. may be at least one of X, Y or Z.Thus, such conjunctive language is not generally intended to imply thatcertain embodiments require at least one of X, at least one of Y and atleast one of Z to each be present.

Similarly, while operations may be depicted in the drawings in aparticular order, it is to be recognized that such operations need notbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. Further, the drawings may schematically depict one more exampleprocesses in the form of a flowchart. However, other operations that arenot depicted can be incorporated in the example methods and processesthat are schematically illustrated. For example, one or more additionaloperations can be performed before, after, simultaneously, or betweenany of the illustrated operations. Additionally, the operations may berearranged or reordered in other implementations. In certaincircumstances, multitasking and parallel processing may be advantageous.Moreover, the separation of various system components in theimplementations described above should not be understood as requiringsuch separation in all implementations, and it should be understood thatthe described program components and systems can generally be integratedtogether in a single software product or packaged into multiple softwareproducts. Additionally, other implementations are within the scope ofthe following claims. In some cases, the actions recited in the claimscan be performed in a different order and still achieve desirableresults.

Example Embodiments

1. A method comprising:

displaying a first version of a virtual three-dimensional model via adisplay of a head-wearable device, wherein the first version of thevirtual three-dimensional model comprises a first version of aconstituent part;

requesting data associated with a second version of the virtualthree-dimensional model from a host device, wherein the second versionof the virtual three-dimensional model comprises a second version of theconstituent part;

determining whether the first version of the constituent part requiresan update based on the second version of the constituent part;

in accordance with a determination that the first version of theconstituent part requires an update based on the second version of theconstituent part:

requesting data associated with the second version of the constituentpart from the host device;

displaying the second version of the virtual three-dimensional model viathe display of the head-wearable device; and

in accordance with a determination that the first version of theconstituent part does not require an update based on the second versionof the constituent part:

forgoing requesting data associated with the second version of theconstituent part from the host device.

2. The method of embodiment 1, wherein the constituent part comprisesmesh data.

3. The method of embodiment 1, wherein the constituent part comprisestexture data.

4. The method of embodiment 1, wherein the host device is a server.

5. The method of embodiment 1, wherein the head-wearable device is afirst head-wearable device, and wherein the host device is a secondhead-wearable device.

6. The method of embodiment 1, the method further comprising storing thedata associated with the second version of the constituent part in amemory.

7. The method of embodiment 1, the method further comprisingdecompressing the data associated with the second version of theconstituent part.

8. A system comprising:

a head-wearable device comprising a display;

one or more processors configured to execute a method comprising:

displaying a first version of a virtual three-dimensional model via thedisplay, wherein the first version of the virtual three-dimensionalmodel comprises a first version of a constituent part;

requesting data associated with a second version of the virtualthree-dimensional model from a host device, wherein the second versionof the virtual three-dimensional model comprises a second version of theconstituent part;

determining whether the first version of the constituent part requiresan update based on the second version of the constituent part;

in accordance with a determination that the first version of theconstituent part requires an update based on the second version of theconstituent part:

requesting data associated with the second version of the constituentpart from the host device;

displaying the second version of the virtual three-dimensional model viathe display; and

in accordance with a determination that the first version of theconstituent part does not require an update based on the second versionof the constituent part:

forgoing requesting data associated with the second version of theconstituent part from the host device.

9. The system of embodiment 8, wherein the constituent part comprisesmesh data.

10. The system of embodiment 8, wherein the constituent part comprisestexture data.

11. The system of embodiment 8, wherein the host device is a server.

12. The system of embodiment 8, wherein the head-wearable device is afirst head-wearable device, and wherein the host device is a secondhead-wearable device.

13. The system of embodiment 8, the method further comprising storingthe data associated with the second version of the constituent part in amemory.

14. The system of embodiment 8, the method further comprisingdecompressing the data associated with the second version of theconstituent part.

15. A non-transitory computer-readable medium storing instructions that,when executed by one or more processors, cause the one or moreprocessors to execute a method comprising:

displaying a first version of a virtual three-dimensional model via adisplay of a head-wearable device, wherein the first version of thevirtual three-dimensional model comprises a first version of aconstituent part;

requesting data associated with a second version of the virtualthree-dimensional model from a host device, wherein the second versionof the virtual three-dimensional model comprises a second version of theconstituent part;

determining whether the first version of the constituent part requiresan update based on the second version of the constituent part;

in accordance with a determination that the first version of theconstituent part requires an update based on the second version of theconstituent part:

requesting data associated with the second version of the constituentpart from the host device;

displaying the second version of the virtual three-dimensional model viathe display of the head-wearable device; and

in accordance with a determination that the first version of theconstituent part does not require an update based on the second versionof the constituent part:

forgoing requesting data associated with the second version of theconstituent part from the host device.

16. The non-transitory computer-readable medium of embodiment 15,wherein the constituent part comprises mesh data.

17. The non-transitory computer-readable medium of embodiment 15,wherein the constituent part comprises texture data.

18. The non-transitory computer-readable medium of embodiment 15,wherein the host device is a server.

19. The non-transitory computer-readable medium of embodiment 15,wherein the head-wearable device is a first head-wearable device, andwherein the host device is a second head-wearable device.

20. The non-transitory computer-readable medium of embodiment 15, themethod further comprising storing the data associated with the secondversion of the constituent part in a memory.

21. The non-transitory computer-readable medium of embodiment 15, themethod further comprising decompressing the data associated with thesecond version of the constituent part.

22. A method, the method comprising:

accessing a virtual three-dimensional model stored in a memory;

determining one or more constituent parts of the virtualthree-dimensional model;

storing the one or more constituent parts in one or more arrays, whereinthe one or more constituent parts are stored separate from the virtualthree-dimensional model;

receiving a connection request from a head-wearable device;

sending a list of available constituent parts to the head-wearabledevice;

receiving a constituent part request from the head-wearable device; and

sending the head-wearable device a requested constituent part based onthe constituent part request.

23. The method of embodiment 22, wherein the constituent part comprisesmesh data.

24. The method of embodiment 22, wherein the constituent part comprisestexture data.

25. The method of embodiment 22, the method further comprisingcompressing the one or more constituent parts.

26. The method of embodiment 22, wherein the virtual three-dimensionalmodel is a first virtual three-dimensional model, and wherein the methodfurther comprises generating a copy of the virtual three-dimensionalmodel based on the requested constituent part.

27. The method of embodiment 22, the method further comprisingdisplaying the requested constituent part via a display of thehead-wearable device.

28. The method of embodiment 22, the method further comprising:

creating a placeholder for a new virtual three-dimensional model; and

updating the placeholder for the new virtual three-dimensional modelbased on the requested constituent part.

29. A system, the system comprising:

a head-wearable device;

one or more processors configured to execute a method comprising:

accessing a virtual three-dimensional model stored in a memory;

determining one or more constituent parts of the virtualthree-dimensional model;

storing the one or more constituent parts in one or more arrays, whereinthe one or more constituent parts are stored separate from the virtualthree-dimensional model;

receiving a connection request from a head-wearable device;

sending a list of available constituent parts to the head-wearabledevice;

receiving a constituent part request from the head-wearable device; and

sending the head-wearable device a requested constituent part based onthe constituent part request.

30. The system of embodiment 29, wherein the constituent part comprisesmesh data.

31. The system of embodiment 29, wherein the constituent part comprisestexture data.

32. The system of embodiment 29, the method further comprisingcompressing the one or more constituent parts.

33. The system of embodiment 29, wherein the virtual three-dimensionalmodel is a first virtual three-dimensional model, and wherein the methodfurther comprises generating a copy of the virtual three-dimensionalmodel based on the requested constituent part.

34. The system of embodiment 29, the method further comprisingdisplaying the requested constituent part via a display of thehead-wearable device.

35. The system of embodiment 29, the method further comprising:

creating a placeholder for a new virtual three-dimensional model; and

updating the placeholder for the new virtual three-dimensional modelbased on the requested constituent part.

36. A non-transitory computer-readable medium storing instructions that,when executed by one or more processors, cause the one or moreprocessors to execute a method comprising:

accessing a virtual three-dimensional model stored in a memory;

determining one or more constituent parts of the virtualthree-dimensional model;

storing the one or more constituent parts in one or more arrays, whereinthe one or more constituent parts are stored separate from the virtualthree-dimensional model;

receiving a connection request from a head-wearable device;

sending a list of available constituent parts to the head-wearabledevice;

receiving a constituent part request from the head-wearable device; and

sending the head-wearable device a requested constituent part based onthe constituent part request.

37. The non-transitory computer-readable medium of embodiment 36,wherein the constituent part comprises mesh data.

38. The non-transitory computer-readable medium of embodiment 36,wherein the constituent part comprises texture data.

39. The non-transitory computer-readable medium of embodiment 36, themethod further comprising compressing the one or more constituent parts.

40. The non-transitory computer-readable medium of embodiment 36,wherein the virtual three-dimensional model is a first virtualthree-dimensional model, and wherein the method further comprisesgenerating a copy of the virtual three-dimensional model based on therequested constituent part.

41. The non-transitory computer-readable medium of embodiment 36, themethod further comprising displaying the requested constituent part viaa display of the head-wearable device.

42. The non-transitory computer-readable medium of embodiment 36, themethod further comprising:

creating a placeholder for a new virtual three-dimensional model; and

updating the placeholder for the new virtual three-dimensional modelbased on the requested constituent part.

43. A system comprising:

a host computing system;

a client computing system comprising a head-wearable display system;

wherein the host computing system comprises one or more processorsconfigured to execute a method comprising:

accessing a virtual three-dimensional model stored in a memory;

decomposing the three-dimensional model into one or more constituentparts;

sending a list of the one or more constituent parts to the clientcomputing system;

receiving a constituent part request from the client computing system;

sending one or more constituent parts that correspond to the constituentpart request to the client computing system; and

wherein the client computing system comprises one or more processorsconfigured to execute a method comprising:

receiving the list of the one or more constituent parts from the hostcomputing system;

sending the constituent part request to the host computing system;

receiving the one or more of the constituent parts that correspond tothe constituent part request from the host computing system;

composing a copy of the virtual three-dimensional model from the one ormore of the constituent parts that correspond to the constituent partrequest.

44. The system of embodiment 43, wherein the constituent part comprisesmesh data.

45. The system of embodiment 43, wherein the constituent part comprisestexture data.

46. The system of embodiment 43, wherein the host computing system is aserver.

47. The system of embodiment 43, wherein the host computing systemcomprises a head-wearable display system.

48. The system of embodiment 43, the method further comprising storingthe one or more constituent parts in a memory.

49. The system of embodiment 43, the method further comprisingdecompressing the one or more constituent parts.

50. A method comprising:

accessing a virtual three-dimensional model stored in a memory;

decomposing the three-dimensional model into one or more constituentparts;

sending a list of the one or more constituent parts to a clientcomputing system comprising a head-wearable display;

receiving a constituent part request from the client computing system;

sending one or more constituent parts that correspond to the constituentpart request to the client computing system;

receiving the list of the one or more constituent parts from a hostcomputing system;

sending the constituent part request to the host computing system;

receiving the one or more of the constituent parts that correspond tothe constituent part request from the host computing system; and

composing a copy of the virtual three-dimensional model from the one ormore of the constituent parts that correspond to the constituent partrequest.

51. The method of embodiment 50, wherein the constituent part comprisesmesh data.

52. The method of embodiment 50, wherein the constituent part comprisestexture data.

53. The method of embodiment 50, wherein the host computing system is aserver.

54. The method of embodiment 50, wherein the host computing systemcomprises a head-wearable display system.

55. The method of embodiment 50, the method further comprising storingthe one or more constituent parts in a memory.

56. The method of embodiment 50, the method further comprisingdecompressing the one or more constituent parts.

57. A non-transitory computer-readable medium storing one or moreinstructions, which, when executed by one or more processors, cause theone or more processors to perform a method comprising:

accessing a virtual three-dimensional model stored in a memory;

decomposing the three-dimensional model into one or more constituentparts;

sending a list of the one or more constituent parts to a clientcomputing system comprising a head-wearable display;

receiving a constituent part request from the client computing system;

sending one or more constituent parts that correspond to the constituentpart request to the client computing system;

receiving the list of the one or more constituent parts from a hostcomputing system;

sending the constituent part request to the host computing system;

receiving the one or more of the constituent parts that correspond tothe constituent part request from the host computing system; and

composing a copy of the virtual three-dimensional model from the one ormore of the constituent parts that correspond to the constituent partrequest.

58. The non-transitory computer-readable medium of embodiment 57,wherein the constituent part comprises mesh data.

59. The non-transitory computer-readable medium of embodiment 57,wherein the constituent part comprises texture data.

60. The non-transitory computer-readable medium of embodiment 57,wherein the host computing system is a server.

61. The non-transitory computer-readable medium of embodiment 57,wherein the host computing system comprises a head-wearable displaysystem.

62. The non-transitory computer-readable medium of embodiment 57, themethod further comprising storing the one or more constituent parts in amemory.

63. The non-transitory computer-readable medium of embodiment 57, themethod further comprising decompressing the one or more constituentparts.

64. A system comprising:

a host computing system, wherein the host computing system comprises oneor more processors configured to execute a method comprising:

accessing a first virtual three-dimensional model via a host assetsmodule;

identifying one or more constituent parts via a decomposition module;

copying the one or more constituent parts to a library via thedecomposition module;

accessing the one or more constituent parts via a host transmissionmodule;

sending the one or more constituent parts to a client computing systemvia the host transmission module; and

the client computing system, wherein the client computing systemcomprises a head-wearable display system, and wherein the clientcomputing system comprises one or more processors configured to executea method comprising:

receiving the one or more constituent parts from the host computingsystem via a client transmission module;

creating an empty object via a composition module;

adding one or more data types to the empty object via the compositionmodule;

adding one or more constituent parts to the empty object via thecomposition module;

storing a copy of the virtual three-dimensional model via a clientassets module; and

wherein the host computing system and client computing system arecommunicably connected via a communication link.

65. The system of embodiment 64, wherein the constituent part comprisesmesh data.

66. The system of embodiment 64, wherein the constituent part comprisestexture data.

67. The system of embodiment 64, wherein the host computing system is aserver.

68. The system of embodiment 64, wherein the host computing systemcomprises a head-wearable display system.

69. The system of embodiment 64, the method further comprising storingthe one or more constituent parts in a memory.

70. The system of embodiment 64, the method further comprisingdecompressing the one or more constituent parts.

71. A method comprising:

at a host computing system, wherein the host computing system comprisesone or more processors configured to execute a method comprising:

accessing a first virtual three-dimensional model via a host assetsmodule;

identifying one or more constituent parts via a decomposition module;

copying the one or more constituent parts to a library via thedecomposition module;

accessing the one or more constituent parts via a host transmissionmodule;

sending the one or more constituent parts to a client computing systemvia the host transmission module; and

at the client computing system, wherein the client computing systemcomprises a head-wearable display system, and wherein the clientcomputing system comprises one or more processors configured to executea method comprising:

receiving the one or more constituent parts from the host computingsystem via a client transmission module;

creating an empty object via a composition module;

adding one or more data types to the empty object via the compositionmodule;

adding one or more constituent parts to the empty object via thecomposition module;

storing a copy of the virtual three-dimensional model via a clientassets module; and

wherein the host computing system and client computing system arecommunicably connected via a communication link.

72. The method of embodiment 71, wherein the constituent part comprisesmesh data.

73. The method of embodiment 71, wherein the constituent part comprisestexture data.

74. The method of embodiment 71, wherein the host computing system is aserver.

75. The method of embodiment 71, wherein the host computing systemcomprises a head-wearable display system.

76. The method of embodiment 71, the method further comprising storingthe one or more constituent parts in a memory.

77. The method of embodiment 71, the method further comprisingdecompressing the one or more constituent parts.

78. A non-transitory computer-readable medium storing one or moreinstructions, which, when executed by one or more processors, cause theprocessors to perform a method comprising:

at a host computing system:

accessing a first virtual three-dimensional model via a host assetsmodule;

identifying one or more constituent parts via a decomposition module;

copying the one or more constituent parts to a library via thedecomposition module;

accessing the one or more constituent parts via a host transmissionmodule;

sending the one or more constituent parts to a client computing systemvia the host transmission module; and

at the client computing system, wherein the client computing systemcomprises a head-wearable display system:

receiving the one or more constituent parts from the host computingsystem via a client transmission module;

creating an empty object via a composition module;

adding one or more data types to the empty object via the compositionmodule;

adding one or more constituent parts to the empty object via thecomposition module;

storing a copy of the virtual three-dimensional model via a clientassets module; and

wherein the host computing system and client computing system arecommunicably connected via a communication link.

79. The non-transitory computer-readable medium of embodiment 78,wherein the constituent part comprises mesh data.

80. The non-transitory computer-readable medium of embodiment 78,wherein the constituent part comprises texture data.

81. The non-transitory computer-readable medium of embodiment 78,wherein the host computing system is a server.

82. The non-transitory computer-readable medium of embodiment 78,wherein the host computing system comprises a head-wearable displaysystem.

83. The non-transitory computer-readable medium of embodiment 78, themethod further comprising storing the one or more constituent parts in amemory.

84. The non-transitory computer-readable medium of embodiment 78, themethod further comprising decompressing the one or more constituentparts.

85. A system comprising:

a host computing system, wherein the host computing system comprises oneor more processors configured to execute a method comprising:

accessing a first virtual three-dimensional model via a load module;

identifying one or more constituent parts via the host library managermodule;

copying the one or more constituent parts to a library via the hostlibrary manager module;

storing the first virtual three-dimensional model via a host worldmodule;

rendering the first virtual three-dimensional model via a host renderpath module;

accessing the one or more constituent parts via a server module;

sending the one or more constituent parts to a client computing systemvia the server module; and

the client computing system, wherein the client computing systemcomprises a head-wearable display system, and wherein the clientcomputing system comprises one or more processors configured to executea method comprising:

receiving the one or more constituent parts from the host computingsystem via a client module;

generating a copy of the virtual three-dimensional model, whereingenerating the copy of the virtual three-dimensional model comprises:

creating an empty object via a client library manager module; and

adding one or more constituent parts to the empty object via the clientlibrary manager module;

storing the copy of the virtual three-dimensional model via a clientworld module;

rendering the copy of the virtual three-dimensional model via a clientrender path module;

wherein the host computing system and client computing system arecommunicably connected via a communication link.

86. The system of embodiment 85, wherein the constituent part comprisesmesh data.

87. The system of embodiment 85, wherein the constituent part comprisestexture data.

88. The system of embodiment 85, wherein the host computing system is aserver.

89. The system of embodiment 85, wherein the host computing systemcomprises a head-wearable display system.

90. The system of embodiment 85, the method further comprising storingthe one or more constituent parts in a memory.

91. The system of embodiment 85, the method further comprisingdecompressing the one or more constituent parts.

92. A method comprising:

at a host computing system:

accessing a first virtual three-dimensional model via a load module;

identifying one or more constituent parts via the host library managermodule;

copying the one or more constituent parts to a library via the hostlibrary manager module;

storing the first virtual three-dimensional model via a host worldmodule;

rendering the first virtual three-dimensional model via a host renderpath module;

accessing the one or more constituent parts via a server module;

sending the one or more constituent parts to a client computing systemvia the server module; and

at the client computing system, wherein the client computing systemcomprises a head-wearable display system:

receiving the one or more constituent parts from the host computingsystem via a client module;

generating a copy of the virtual three-dimensional model, whereingenerating the copy of the virtual three-dimensional model comprises:

creating an empty object via a client library manager module; and

adding one or more constituent parts to the empty object via the clientlibrary manager module;

storing the copy of the virtual three-dimensional model via a clientworld module;

rendering the copy of the virtual three-dimensional model via a clientrender path module;

wherein the host computing system and client computing system arecommunicably connected via a communication link.

93. The method of embodiment 92, wherein the constituent part comprisesmesh data.

94. The method of embodiment 92, wherein the constituent part comprisestexture data.

95. The method of embodiment 92, wherein the host computing system is aserver.

96. The method of embodiment 92, wherein the host computing systemcomprises a head-wearable display system.

97. The method of embodiment 92, the method further comprising storingthe one or more constituent parts in a memory.

98. The method of embodiment 92, the method further comprisingdecompressing the one or more constituent parts.

99. A non-transitory computer-readable medium storing one or moreinstructions, which, when executed by one or more processors, cause theone or more processors to perform a method comprising:

a host computing system, wherein the host computing system comprises oneor more processors configured to execute a method comprising:

accessing a first virtual three-dimensional model via a load module;

identifying one or more constituent parts via the host library managermodule;

copying the one or more constituent parts to a library via the hostlibrary manager module;

storing the first virtual three-dimensional model via a host worldmodule;

rendering the first virtual three-dimensional model via a host renderpath module;

accessing the one or more constituent parts via a server module;

sending the one or more constituent parts to a client computing systemvia the server module; and

the client computing system, wherein the client computing systemcomprises a head-wearable display system, and wherein the clientcomputing system comprises one or more processors configured to executea method comprising:

receiving the one or more constituent parts from the host computingsystem via a client module;

generating a copy of the virtual three-dimensional model, whereingenerating the copy of the virtual three-dimensional model comprises:

creating an empty object via a client library manager module; and

adding one or more constituent parts to the empty object via the clientlibrary manager module;

storing the copy of the virtual three-dimensional model via a clientworld module;

rendering the copy of the virtual three-dimensional model via a clientrender path module;

wherein the host computing system and client computing system arecommunicably connected via a communication link.

100. The non-transitory computer-readable medium of embodiment 99,wherein the constituent part comprises mesh data.

101. The non-transitory computer-readable medium of embodiment 99,wherein the constituent part comprises texture data.

103. The non-transitory computer-readable medium of embodiment 99,wherein the host computing system is a server.

104. The non-transitory computer-readable medium of embodiment 99,wherein the host computing system comprises a head-wearable displaysystem.

105. The non-transitory computer-readable medium of embodiment 99, themethod further comprising storing the one or more constituent parts in amemory.

106. The non-transitory computer-readable medium of embodiment 99, themethod further comprising decompressing the one or more constituentparts.

107. A system comprising:

a non-transitory computer-readable memory storing a three-dimensionalmodel comprising one or more constituent parts, and

a host computing system comprising one or more processors configured toaccess, from the non-transitory computer-readable memory, thethree-dimensional model, and further configured execute a methodcomprising:

storing one of the one or more constituent parts of thethree-dimensional model into an array stored in the non-transitorycomputer-readable memory,

packaging the array into a package in a library of the host computingsystem.

108. The system of embodiment 107, wherein the constituent partcomprises mesh data.

109. The system of embodiment 107, wherein the constituent partcomprises texture data.

110. The system of embodiment 107, wherein the host computing system isa server.

111. The system of embodiment 107, wherein the host computing systemcomprises a head-wearable display system.

112. The system of embodiment 107, the method further comprising storingthe one or more constituent parts in a memory.

113. The system of embodiment 107, the method further comprisingdecompressing the one or more constituent parts.

114. A method comprising:

storing, via a non-transitory computer-readable memory, athree-dimensional model comprising one or more constituent parts;

accessing, from the non-transitory computer-readable memory, thethree-dimensional model;

storing one of the one or more constituent parts of thethree-dimensional model into an array stored in the non-transitorycomputer-readable memory,

packaging the array into a package in a library of the host computingsystem.

115. The method of embodiment 114, wherein the constituent partcomprises mesh data.

116. The method of embodiment 114, wherein the constituent partcomprises texture data.

117. The method of embodiment 114, wherein the non-transitorycomputer-readable memory is part of a server.

118. The method of embodiment 114, wherein the non-transitorycomputer-readable memory is part of a head-wearable display system.

119. The method of embodiment 114, the method further comprising storingthe one or more constituent parts in a memory.

120. The method of embodiment 114, the method further comprisingdecompressing the one or more constituent parts.

121. A non-transitory computer-readable medium storing one or moreinstructions, which, when executed by one or more processors, cause theone or more processors to perform a method comprising:

storing, via a non-transitory computer-readable memory, athree-dimensional model comprising one or more constituent parts;

accessing, from the non-transitory computer-readable memory, thethree-dimensional model;

storing one of the one or more constituent parts of thethree-dimensional model into an array stored in the non-transitorycomputer-readable memory,

packaging the array into a package in a library of the host computingsystem.

122. The non-transitory computer-readable medium of embodiment 121,wherein the constituent part comprises mesh data.

123. The non-transitory computer-readable medium of embodiment 121,wherein the constituent part comprises texture data.

124. The non-transitory computer-readable medium of embodiment 121,wherein the non-transitory computer-readable memory is part of a server.

125. The non-transitory computer-readable medium of embodiment 121,wherein the non-transitory computer-readable memory is part of ahead-wearable display system.

126. The non-transitory computer-readable medium of embodiment 121, themethod further comprising storing the one or more constituent parts in amemory.

127. The non-transitory computer-readable medium of embodiment 121, themethod further comprising decompressing the one or more constituentparts.

Examples of constituent parts may be geometric data, material data,vertex tables, one or more textures, triangle indices, or any other dataused to ultimately define a full representation of a 3D model.

1. A system comprising: one or more processors configured to execute amethod comprising: accessing a virtual three-dimensional model stored ina memory; decomposing the virtual three-dimensional model intoconstituent parts, wherein the constituent parts comprise mesh data,texture data, or a combination thereof; determining whether a clientcomputing system is in communication with a host computing system; inaccordance with a determination that the client computing system is incommunication with the host computing system, sending, to the clientcomputing system, a library comprising the constituent parts, thelibrary corresponding to the virtual three-dimensional model; receivinga constituent part request from the client computing system; and sendingone or more of the constituent parts that correspond to the constituentpart request to the client computing system, wherein the clientcomputing system is configured to compose a view of the virtualthree-dimensional model based on the one or more of the constituentparts.
 2. The system of claim 1, wherein the system comprises the hostcomputing system.
 3. The system of claim 1, further comprising ahead-wearable display system.
 4. The system of claim 1, the methodfurther comprising storing the one or more of the constituent parts inthe memory.
 5. The system of claim 1, the method further comprisingcompressing the one or more of the constituent parts.
 6. The system ofclaim 1, wherein a physically integrated device comprises the system andfurther comprises the client computing system.
 7. The system of claim 1,wherein determining whether the client computing system is incommunication with the system comprises: receiving a connection requestfrom the client computing system; and accepting the connection requestfrom the client computing system.
 8. The system of claim 1, wherein themethod further comprises: receiving a library of second constituentparts from the second host computing system, the library of secondconstituent parts comprising second constituent parts corresponding to asecond virtual three-dimensional model, wherein the second constituentparts comprise second mesh data, second texture data, or a combinationthereof; determining one or more of the second constituent parts to beincluded in a second constituent part request based on a comparison ofthe library of second constituent parts and the library of constituentparts; sending the second constituent part request to the second hostcomputing system; receiving the one or more of the second constituentparts that correspond to the second constituent part request from thesecond host computing system.
 9. A system comprising: one or moreprocessors configured to execute a method comprising: receiving alibrary of constituent parts from a host computing system, the libraryof constituent parts comprising constituent parts corresponding to avirtual three-dimensional model, wherein the constituent parts comprisemesh data, texture data, or a combination thereof; determining one ormore parts to be included in a constituent part request based on acomparison of the library of the constituent parts and a client libraryof constituent parts; sending the constituent part request to the hostcomputing system; receiving one or more of the constituent parts thatcorrespond to the constituent part request from the host computingsystem; and composing a view of the virtual three-dimensional modelbased on the received one or more of the constituent parts.
 10. Thesystem of claim 9, wherein the system comprises a head-wearable displaysystem.
 11. The system of claim 9, the method further comprising storingthe one or more of the constituent parts in a memory.
 12. The system ofclaim 9, the method further comprising decompressing the one or more ofthe constituent parts.
 13. The system of claim 9, wherein a physicallyintegrated device comprises the system and further comprises the hostcomputing system.
 14. The system of claim 9, the wherein determining theone or more parts to be included in the constituent part request isfurther based on a version number associated with the one or more parts.15. The system of claim 9, the method further comprising: sending aconnection request to the host computing system; and connecting to thehost computing system in accordance with receiving an indication of anacceptance of the connection request from the host computing system. 16.The system of claim 9, wherein the method further comprises: accessingthe virtual three-dimensional model stored in a memory; decomposing thevirtual three-dimensional model into second constituent parts, whereinthe second constituent parts comprise second mesh data, second texturedata, or a combination thereof; determining whether the system is incommunication with a client computing system; in accordance with adetermination that the system is in communication with the clientcomputing system, sending, to the client computing system, the clientlibrary of constituent parts, the client library corresponding to thevirtual three-dimensional model; receiving a second constituent partrequest from the second client computing system; and sending one or moreof the second constituent parts that correspond to the secondconstituent part request to the client computing system, wherein theclient computing system is configured to compose a view of the virtualthree-dimensional model based on the one or more of the secondconstituent parts.